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D epartment of Labor. The views expressed herein do not necessarily reflect the policies or 
opinions of the U.S. D epartment of Labor. 




PAGE IS INTENTIONALLY LEET BLANKTO ALLOW EOR DOUBLE-SIDED COPYING 




Contents 



Page 

Executive Summary vii 

Introduction 1 

The Q 0 P Targ et G roup 5 

The Program Model 6 

How Well WasQOP Implemented? 9 

How Much Did QOP Cost? 12 

How Much Did Enrollees Participate in QOP? 13 

Estimating the Impacts oe QOP 16 

IMPACTSON QOP'SEIRST PRIMARY OUTCOME: HIGH SCHOOL COMPLETION 19 

IMPACTSON QOP's S econd Primary Outcome: Postsecondary Education or 
Training 21 

IMPACTSON Employment AND Earnings 25 

IMPACTSON QOP's Secondary Outcomes: High School Pereormance and 
Risky Behaviors 28 

IMPACTSON Subgroups 33 

Impacts by Sex 34 

Impacts by Age When E ntering N inth G rade 40 

Impacts by Rank in the Basehne G rade D istribution 46 

Impacts BY Site 52 

Summary oe Impacts 64 

Discussion 67 

Why D idn't Q 0 P Have Larger Impacts 0 veraH? 67 

Why Were the Results in the Demonstration Different from Those of the Pilot 

Study? 69 

Why Were There Impacts for Some G roups but N ot for 0 thers? 72 

What Lessons D oes This Provide for the Future? 73 



V 




Contents 



References 



76 



Appendix A: 

Appendix B: 
Appendix C: 
Appendix D: 
Appendix E: 
Appendix F: 
Appendix G: 
Appendix H: 



Obtaining an Evaluation Sample and Conducting Random Assignment 
The Baseline Data 

Follow-up Data from the Third Telephone Survey 
Outgo MESAND Subgroups 

Weighting, Impact Estimation, and Variance Estimation 
Sensitivity Analyses 

QOP AND Control Group Means for Subgroups 
QOP AND CONTROLGROUP MEANS FOR SITES 



VI 




Executive Summary 



From July 1995 through September 2001, the U.S. D epartment of Labor (D 0 L) and the 
Ford Foundation (Ford) operated a demonstration of the Quantum Opportunity Program 
(QOP). QOP offered intensive and comprehensive services to help at-risk youth graduate 
from high school and enroU in postsecondaiy education or training. 

The QOP demonstration targeted youth with low grades entering high schools with 
high dropout rates. Randomly selected eligible youth were enrolled in Q 0 P and served even 
if they transferred to other schools, dropped out of school, became incarcerated, or became 
inactive in QOP for a long time. QOP's piimaiy goals were to increase the rates of high 
school graduation and enrollment in postsecondaiy education or training. Its secondary goals 
were to improve high school grades and achievement test scores and to reduce risky 
behaviors, such as substance abuse, crime, and teen parenting. 

QOP was mainly an after-school program providing case management and mentoring, 
supplemental education, development^ activities, community service activities, supportive 
services, and financial incentives. These services were provided year-round for five years to 
enrollees who had not graduated from high school, and were designed to be comprehensive 
enough to address all barriers to success and to be intensive. The program model specified 
roughly 15 to 25 enrollees per case manager, and it prescribed an annual participation goal of 
750 hours for each enroUee who had not graduated. From graduation to the end of the 
demonstration, enrollees who had graduated received limited services— some mentoring and 
assistance with enrolling in postsecondaiy education or training. 

Community-based organizations (CBOs) in seven sites operated QOP demonstration 
programs. Five sites (Cleveland, Fort Worth, Houston, Memphis, and Washington, DC) 
were funded by D 0 L. Four of the five served 100 youth each, and the Washington, D C site 
served 80 youth. The other two sites (Philadelphia and Y akima) served 50 youth each with 
funding from Ford, which also funded the technical assistance provided to sites throughout 
the demonstration. D 0 L has funded the evaluation of the Q 0 P demonstration. 

E valuation D esign 

To estimate QOP's impacts on high school performance and graduation, postsecondaiy 
education or training, and risky behaviors, we have conducted four surveys, administered 
achievement tests in reading and mathematics, and collected high school transcripts for a 
group of youth who were enrolled in QOP and a group of statistically identical youth— the 
control group— who were not allowed to participate in QOP. We formed the QOP and 
control groups at the start of the demonstration by randomly assigning each of the nearly 
1,100 youth eligible for the program to one group or the other. 

In this report, we present Q 0 P's impacts on outcomes measured using data from the 
fourth— and final— survey of the evaluation. The survey was administered by telephone and 
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began nearly six years after most sample members were scheduled to graduate from high 
school, a time when they were 23 to 25 years old (nearly five years after most sample 
members were scheduled to graduate from high school, at a time when they were 22 to 24 
years old in the Washington, D C site, where program operations began one year later than in 
the other sites). 

Findings based on data from the first two evaluation surveys— which were conducted 
in-person and by telephone, respectively— were presented in previous reports (Maxfield et al. 
2003b and Schirm et al. 2003). Those surveys were administered during the fourth and fifth 
years of the demonstration, that is, before the demonstration was over and when many 
sample members were still attending high school. The first post-intervention impacts were 
presented in a subseguent report (Schirm and Rodriguez- Planas 2004). They were estimated 
from data collected in the third evaluation survey— a telephone survey— that began a little 
more than three years after sample members were scheduled to graduate from high school 
(two years after scheduled graduation in the Washington, D C site). 

The Context for Interpreting the Impacts of the Q OP Demonstration 

The impacts of the QOP demonstration are not determined entirely by the features of 
the QOP model. The impacts are also influenced— perhaps heavily— by how well the 
demonstration sites implemented the QOP model, how much they spent on the program, 
and the extent to which QOP enroUees participated in the program. Because the guality of 
implementation, the amount of spending, and the extent of participation were not varied by 
design, it is not valid to conclude, for example, that better impacts in one site relative to 
other sites were caused by closer fidehty to the QOP model in that site. However, 
understanding the patterns of implementation, costs, and participation provides a context 
for assessing the impacts presented in this report and understanding the potential sources of 
variation in impacts. 

Through annual site visits, annual QOP conferences, and conference calls with QOP 
staff, we assessed how weU the CBOs in the QOP demonstration implemented the program 
model. From information provided by QOP staff, we also measured QOP costs and the 
extent to which enrollees participated in QOP's educational, developmental, and community 
service activities. Because financial incentives were provided for participation in these three 
activities only, the participation data do not include time spent being mentored if the 
mentoring was not part of an educational, developmental, or community service activity. 

Although all sites were encouraged to implement the Q 0 P model, neither DOT nor, to 
a lesser degree. Ford, reguired sites to implement fully all of the elements of the QOP 
model, in part to allow some flexibihty for adjusting implementation to local or changing 
circumstances. Our analysis of program implementation revealed that two sites 
implemented a version of Q 0 P that deviated substantially from the program model and that 
the other five sites implemented versions that deviated moderately from the model. With 
the exception of the Philadelphia site— where the program was operated by the CBO that 
helped to design the QOP model and oversaw a small-scale pilot of QOP from 1989 
through 1993— local CBOs found implementing QOP difficult, primarily because QOP was 
substantially more comprehensive, intensive, and complex than their traditional programs. 
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Although sites implemented the mentoring and developmental components relatively well, 
no site fully and effectively implemented the education component, and sites generally did 
not meet their enrollees' needs for some supportive services, including child care, health and 
mental health services, and substance abuse treatment. 

In addition to the deviations from the program model, we found that most enrollees 
attended fewer program activities than was stipulated by the participation goal of 750 hours 
per year. Enrollees spent an average of 177 hours per year on QOP's educational, 
developmental, and community service activities— 24 percent of the annual goal of 750 
hours— through the first four years of the demonstration. The average fell from 247 hours 
in the first year to 103 hours in the fourth year, while the fraction of enrollees spending no 
time at aH on these activities rose from 1 percent to 26 percent. We also found that 
participation varied substantially from site to site, ranging from a low of 68 hours per year to 
a high of 345 hours per year. 

The total cost of QOP per enroHee over the full five-year demonstration period was 
$18,000 to $22,000 for D 0 L-funded sites; $23,000 for the Yakima site; and $49,000 for the 
Philadelphia site. These figures do not include the cost of the technical assistance that was 
provided to sites. 

What Were Q 0 P's Impacts? 

Overall, we find that QOP did not achieve its primary or secondary objectives. 
However, the lack of overall impacts masks some suggestive evidence of promising effects 
for particular types of students. Although our findings are not conclusive, we find some 
beneficial effects for the approximately two -thirds of enroUees who were age 14 or younger 
when they entered ninth grade as well as for enroUees in the Cleveland, Philadelphia, and 
Washington, D C sites. In contrast, we find almost no beneficial effects for enrollees in the 
other four sites. 

Impacts on Primary Outcomes 

• QOP did not achieve its first primary objective. That is, it did not increase 
the UkeUhood of graduating from high school with a diploma. It also did not 
increase the likelihood of completing high school by earning either a diploma or 
aGED. 

• QOP has not achieved its second primary objective. It has not increased 
the likeUhood of ever engaging in postsecondaiy education or training, including 
coUege, vocational/ technical school, an apprenticeship, or the mUitaiy. 
Eurthermore, QO P has not increased persistence in such activities and, thereby, 
attainment of postsecondaiy education or training. Although data collect^ 
earlier in the evaluation indicated that QOP was increasing rates of entry into 
postsecondaiy education or training when sample members were in their late 
teens and early twenties (Maxfield et al. 2003b and Schirm and Rodriguez- Planas 
2004), the most recently coUected data reveal that this impact was not sustained 
in the longer run as sample members entered their mid-twenties. 
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Impacts on Employment and E amings 

• QOP has not improved employment-related outcomes. Improving such 
outcomes was a principal motivation for Q 0 P's two primaiy objectives and, 
thus, an implicit, long-run goal of the program. However, when sample 
members were entering their mid-twenties, QOP had not increased the 
likehhood of being employed, the fraction of time employed, annual or hourly 
earnings, or the likelihood of having a job with benefits, such as health 
insurance, paid time off, or pension and retirement benefits. 

Impacts on Secondary Outcomes 

• QOP did not achieve its secondary objective of improving high school 
grades and achievement test scores. This finding, presented in previous 
reports, is based on data from transcripts and reading and mathematics tests 
administered for the evaluation. 

• QOP has not generally achieved its secondary objective of reducing the 
broad range of risky behaviors targeted by the program. When sample 
members were in their late teens, Q 0 P did not reduce any risky behaviors, such 
as binge drinking, illegal drug use, crime, or teen parenting (Maxfield et al. 
2003b). Although it still did not reduce binge drinking or crime when enrollees 
were in their early twenties, QOP did reduce illegal drug use (Schirm and 
Rodriguez-Planas 2004). More recently, according to data collected when 
sample members were entering their mid- twenties, Q 0 P has not had any such 
beneficial effects in reducing substance abuse, but has had detrimental effects 
on crime and involvement with the criminal justice system, increasing by 3 
percentage points the likelihood of committing a crime in the three months 
prior to the most recent survey and by 6 percentage points the likelihood of 
being arrested for or charged with a crime in the two years before the survey. 

Subgroup and Site Impacts 

• QOP seems to have been more effective for younger enrollees than for 
older enrollees. QOP increased rates of high school completion and 
engagement in postsecondaiy education or training among younger enrollees 
(the two-thirds of enrollees who were age 14 or younger when they entered the 
ninth grade), but it had no such impacts on older enrollees (those who were 
over age 14 when they entered the ninth grade). For younger enrollees, QOP 
increased by 7 percentage points the likelihood of receiving a diploma and by 6 
percentage points the likehhood of receiving a diploma or GED. It also 
increased by 10 percentage points both the likehhood of ever engaging in 
postsecondaiy education or training and the likehhood of completing at least 
two years of college or the mhitaiy, completing vocational/ technical school or 
an apprenticeship, or being honorably discharged from the mhitaiy. Q 0 P did 
not have consistent patterns of impacts across subgroups defined by the other 
two observed basehne characteristics: sex or rank in the eligible grade 
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distribution. (The eligible grade distribution was based on grade point average 
in the eighth grade and excluded youth who were inehgible for Q 0 P because 
their grades were too high.) 

• QOP's impacts varied by site. The Cleveland, Philadelphia, and Washington, 
D C sites had mostly beneficial impacts. The Cleveland site was the only site 
that increased high school completion, raising the hkehhood of earning a 
diploma or G ED by 19 percentage points. The Cleveland site also increased the 
likehhood of ever attending a two- or four-year college by 18 percentage points 
and the likelihood of earning a bachelor's degree by 6 percentage points. In 
addition to beneficial impacts on some employment-related outcomes and on 
smoking and binge drinking rates, the Cleveland site reduced by 19 percentage 
points the likelihood of receiving welfare or food stamps. A 13-percentage- 
point increase in the hkelihood of committing a crime in the two years before 
the most recent survey was the Cleveland site's only detrimental impact. The 
Philadelphia site had beneficial impacts on postsecondaiy educational 
attainment, increasing by 18 percentage points the likehhood of ever attending a 
four-year college and by 13 percentage points the likehhood of completing at 
least two years at a four-year college. Although the Philadelphia site had 
detrimental impacts on the rate of frequent binge drinking and the hkelihood of 
being arrested or charged with a crime in the two years before the most recent 
survey, it reduced by 23 percentage points the likehhood of receiving welfare or 
food stamps. The Washington, DC site increased postsecondaiy education or 
training, raising by 15 percentage points the hkelihood of ever engaging in 
postsecondaiy education or training and by 19 percentage points the likehhood 
of completing at least two years of college or mhitaiy service, completing 
vocational/ technical school or an apprenticeship, or being honorably discharged 
from the mhitaiy. The Washington, DC site also had beneficial impacts on the 
rate of frequent binge drinking and the likehhood of having a child with whom 
the enrohee was not hving. With the exception of a decrease in the proportion 
of enrohees with poor self-reported health status in the Memphis site, none of 
the other four sites— Fort Worth, Houston, Memphis, or Yakima— had 
beneficial impacts, whhe some had detrimental impacts. 
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Introduction 



This report presents the final estimated impacts of the Quantum Opportunity Program 
(QOP)^ demonstration.^ From July 1995 through September 2001, the U.S. Department of 
Labor (DOL) and the Ford Foundation (Ford) operated a demonstration of QOP designed 
to help at-risk^ high-school-age youth graduate from high school and enroU in postsecondaiy 
education or training to improve their prospects for success in the labor market. QOP was 
an intensive case management and mentoring program that emphasized after-school 
supplemental academic education, developmental activities, and community service. 

QOP is one of several approaches to assisting at-risk youth evaluated in recent years by 
DOL and the Department of Education (ED), including Job Corps, Job Training 
Partnership Act (JTPA) youth programs. Career Academies, Center for Employment 
Training (CET), Upward Bound, and Talent Search. As employers demanded more 
advanced technical, cognitive, and work- readiness skills in entiy- level employees, DOL and 
ED became concerned that some youth are not effectively prepared to meet these rising 
standards. Such youth are at increased risk of unemployment, poverty, welfare dependency, 
substance abuse, criminal activity, and teenage childbearing. Finding effective approaches to 
assisting these youth in achieving economic self-sufficiency is critical to avoiding the 
personal losses resulting from such hfe events and to reducing the costs associated with, for 
example, criminal activity and the provision of assistance through Unemployment Insurance, 
Worlrforce Investment Act (WIA), Temporaiy Assistance for Needy Families (TANF), 
Medicaid, and other pubhc programs. The importance to the nation's economy was 
described in a speech by Alan Greenspan, then chair of the Board of Governors of the 
Federal Reserve System: 

As history clearly shows, our economy is best served by full and vigorous 
engagement in the global economy. Consequently, we need to increase our efforts 
to ensure that as many of our citizens as possible have the opportunity to capture 
the benefits that flow from that engagement... . [0]ne criticd element in creating 
those opportunities is to provide rigorous education and ongoing training to aU 
members of our society, ... a strategy that we now should embrace with renewed 
commitment (Greenspan 2004). 



1 The acronym QOP is customarily pronounced kwap. 

2 Maxfield et al. 2003b and Schirm et al. 2003 presented short-term impacts based on data collected while 
the demonstration was stiU underway, and Schirm and Rodriguez-Planas (2004) presented the first post- 
intervention impacts, which were estimated from data collected one to two years after the end of the 
demonstration. 

2 At-risk youth are at a greater risk of substance abuse, criminal activity, teenage childbearing, not 
completing high school, or not enrolling in a postsecondaiy education or training program, compared to the 
average high-school-age youth in the United States. 
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Addressing an issue that is especially relevant to the long-term prospects of at-risk youth, 
Chairman G reenspan also observed: 



[T]he apparent imbalances between the supply and demand for labor across the 
spectrum of skills... have the potential to hamper the adjustment flexibility of our 
economy overall. But these growing imbalances are also aggravating the ineguality 
of incomes in this country. The single central action necessary to amehorate these 
imbalances and their accompanying conseguences for income ineguahty is to boost 
the skills, and thus earning potential, of those workers lower on the skUl ladder 
(G reenspan 2004). 

Recent data confirm the large differences in earnings across education/ skiU levels. In 
2000, males and females age 25 to 34 with at least a bachelor's degree earned on average 60 
and 95 percent more, respectively, than males and females age 25 to 34 who had received a 
high school diploma or general educational development (GED) certificate but had not 
attended college. D espite their earnings disadvantage relative to college graduates, the young 
adults who had completed high school via a diploma or G ED still earned substantially more 
than high school dropouts of the same age— 37 percent more for males and 43 percent more 
for females, on average (U.S. Department of Education 2002a). 

With competition from abroad and the introduction of new technologies, there have 
been some trends in relative earnings over the last two decades— most notably, an increase 
in earnings for college graduates relative to high school graduates (U.S. Department of 
Education 2002a). Nevertheless, substantial gaps between the earnings of young adults vUth 
different levels of educational attainment persisted throughout the period, including the late 
1980s when three organizations— Opportunities Industriahzation Centers of America 
(OICA) in Philadelphia; the Eord Eoundation; and Remediation and Training Institute in 
Alexandria, Virginia— developed the QOP model. The developers of the QOP model 
beheved that acguiring human capital by completing high school and engaging in 
postsecondary education or training substantially enhances a youth's prospects for a 
successful career with sufficiently high earnings to support a good standard of hving for a 
family. They also beheved that engaging in risky behaviors— such as substance abuse, crime, 
and teenage childbearing— created barriers to success. 

To promote the acgirisition of human capital and the avoidance of risky behaviors, the 
developers of the QO P model created the program to provide intensive and comprehensive 
services over several years to a broad range of at-risk youth, including especially those youth 
who might not otherwise be sufficiently motivated to apply to or actively participate in such 
a program. In addition to educational services for developing or encouraging the 
development of human capital, QOP would emphasize mentoring and personal and cultural 
development activities. Prom the perspective of the juvenile justice literature, the mentoring 
and development activities would mitigate the influence of risk factors in a youth's social 
environment— such as gangs and neighborhood drug dealers— and strengthen the youth's 
resihency in resisting the risk factors (U.S. D epartment of Justice 1995). 

Poliowing development of the program model, Eord funded and OICA oversaw a 
small-scale QOP pilot in five sites from 1989 through 1993. The Center for Human 
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Resources at Brandeis University evaluated the phot, obtaining some findings that were 
encouraging to D 0 L and Ford. These findings included increases in high school graduation 
and enrollment in postsecondaiy education as well as reductions in risky behaviors. 
However, the Brandeis evaluation was limited by a small sample size (only 25 Q 0 P enroUees 
in each of five sites), results primarily attributable to one site, and poor implementation in 
most of the other sites. In fact, one site was completely dropped from the analysis because 
of poor implementation (Hahn, Leavitt, and Aaron 1994).^ 

The results from the pilot emerged at a time when D 0 L considered conducting random 
assignment evaluations of intensive youth program models in an effort to identify effective 
programs. Concerned about the conseguences of high dropout and low postsecondaiy 
enrollment rates among urban youth, DOL sought to further evaluate the promising QOP 
model. In early 1995, DOL and Ford agreed to test QOP on a larger scale via a 
demonstration with two sites— Philadelphia and Y akima— under private management and 
administration and with five sites— Cleveland, Fort Worth, Houston, Memphis, and 
Washington, DC— under federal management and administration, specifically, under the 
pilot and demonstration authority of JTPA. 

The QOP demonstration served a single cohort of youth from the beginning of the 
ninth grade in the fall of 1995 through the fall of 2000.^ A local community-based 
organization (CBO) in each of the seven demonstration sites implemented and operated a 
QOP program. Each CBO teamed with from one to three high schools and had 50, 80, or 
100 youth enrolled in the program. By the end of the demonstration, enroUees were in a 
variety of statuses, including attending college or another postsecondaiy training program, 
stiU attending high school, attending a GED certification program, working after finishing 
high school, and working or unemployed after dropping out of high school. 

The primaiy objectives of the demonstration were to increase the likehhood of high 
school completion and the hkehhood of enroUment in postsecondaiy education or training. 
Its secondaiy objectives were to increase academic achievement while in high school and to 
reduce risky behaviors, such as substance abuse, crime, and teenage chUdbeaiing. Under 
contract to D 0 L, Mathematica Policy Research has evaluated the Q 0 P demonstration, and 
assessed in previous reports the program's implementation, short-term impacts, and early 
post-intervention impacts (Maxfield et al. 2003a and 2003b, Schirm et al. 2003, and Schirm 
and Rodriguez-Planas 2004). The short-term impacts were based on data collected during 
the fourth and fifth years of the demonstration, that is, whUe sites were stiU providing 
services to enroUees and when many youth were either stiU attending high school or had only 
recently graduated. The early post-intervention impacts were based on data coUected a Uttle 
more than two years after the end of the demonstration when most members of the 
evaluation sample were 21 or 22 years old (one year after the end of the demonstration in 
the Washington, DC site, when most sample members there were 20 or 21 years old). This 
report presents even longer-term program impacts, which are based on data coUected a little 
more tiian four years after the end of the demonstration, when most members of the 



^ The pilot and differences between it and the demonstration are discussed in more detail below. 
5 All events occurred one year later in the Washington, D C site. 
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evaluation sample were 23 to 25 years old (three years after the end of the demonstration in 
the Washington, D C site, when most sample members there were 22 to 24 years old). 

After briefly describing the QOP target group and program model in the next section, 
we summarize our previously- reported findings pertaining to the following questions: 

• How well was the QOP program model implemented in the demonstration 
sites? 

• How much did Q 0 P cost? 

• How much time did enroHees spend on program activities? 

Following the review of the implementation, cost, and participation findings, we present 
estimates of the impacts of Q 0 P. 
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The QOP Target Group 



The target group in the QOP demonstration was youth entering the ninth grade in fall 
1995 (1996 in the Washington, D C site) who met the following criteria: 

• Began the ninth grade at a high school selected for the QOP demonstration. 
High schools in the DOL-funded sites were reguired to have dropout rates of 
40 percent or more. There was no such explicit criterion for the sites funded by 
the Ford Foundation. 

• Were not repeating the ninth grade. 

• Were not so physically disabled or learning disabled that participation in the 
program would not be appropriate, as determined by the school. 

• Had a grade point average (G PA) below the 67th percentile among the students 
meeting the first three reguirements. (The G PA was calculated from final grades 
received in the eighth grade.) 

With the exception of the Yakima site, the QOP demonstration schools primarily 
served predominantly black or Hispanic populations in urban neighborhoods. The average 
school size, 1,573 students, was typical for urban high schools during the years that QOP 
operated.® Students within a school tended to be homogeneous with respect to 
race/ ethnicity, with enrollments more than 90 percent black at eight schools and more than 
90 percent Hispanic at one school. Of the large urban schools, only the high school in Fort 
Worth served a relatively mixed student population; 11 percent of its students were black 
and 45 percent, Hispanic. Black and Hispanic students constituted 42 percent of the 
enrollment at the demonstration high school located in the mid-size city of Y akima. The 
neighborhoods served by the schools varied in their poverty status, as indicated by the 
percentage of students certified for free lunch.^ In 2003, two schools had fewer than 40 
percent of their students certified for free lunch while six of the schools had a rate higher 
than the national average for large urban high schools (44 percent).® In the Philadelphia high 
school, 90 percent of the students were certified for free lunch in 2003. 



6 The average enrollment m the 1995- 1996 school year at schools spanning grades 9 to 12 and located in 
a large city was 1,663 students (calculations based on Common Core of Data Public Elementary/ Secondary 
School Universe; U.S. Department of Education 2006). 

’ Calculation of percentages of black and Hispanic students, percentages certified for free lunch, and 
average enrollment was based on the Common Core of D ata Public Elementary/ Secondary School Universe 
(http://nces.ed.gov/ ccd/ bat/ , accessed April 11, 2006; U.S. Department of Education 2006). The 2003-2004 
school year is the most recent school year for which a majority of the demonstration schools report lunch 
eligibUily. 

® N 0 schools in T ennessee reported the percentage of students certified for free lunch in 2003- 2004 and 
thus the information is imavahable for the three QOP schools m Memphis. 
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The Program Model 



The QOP model consisted of four primaiy components: case management and 
mentoring, education, developmental activities, and community service. Secondary aspects 
of the program model included financial incentives— stipends, accrual accounts, enroUee 
bonuses, and staff bonuses— and supportive services— snacks, transportation assistance, and 
other services as needed, including child care, health and mental health services, and 
substance abuse treatment. 

Compared to the models for most other youth programs, the QOP model reguired 
more intensive case management and mentoring in four ways: 

1. Enrollees were to have greater access to case managers and were to be involved 
in more program activities for longer periods of time. Each case manager was to 
have a caseload of approximately 15 to 25 enrollees. The QOP model set a 
target of 250 hours per year for activities in each of three service components— 
education, developmental activities, and community service— for a total of 750 
hours per year until an enrollee graduated from high school. Enrollees who took 
full advantage of QOP received services for five years.® Most case managers 
were available during off hours for enroUees to call in emergencies.^® 

2. EnroUees were to remain in the program for longer periods because services 
would be provided for up to five years and program ehgibUity was not contingent 
on enroUee behavior. Youth continued to be enroUed in QOP even if they 
transferred to another school, dropped out of school, became incarcerated, or 
became inactive in QOP for a long time. In contrast to some other youth 
programs, Q 0 P did not accept or retain only those youth who were sufficiently 
motivated to apply and actively participate. The demonstration's approach of 
enroUing aU randomly selected eligible youth reflected the program's philosophy 
that the least-motivated youth might benefit the most from receiving help. 

3. Enrollees were to receive more comprehensive services because the scope of 
case management caUed for addressing aU barriers that enroUed youth faced. 
Case managers either addressed a barrier directly— by arranging transportation to 
program activities, for example— or referred the enroUee to another community 
resource, such as a substance abuse treatment program. 

4. EnroUees were to participate in the program throughout school vacations and 
the summer. EnroUees who faUed a class during the school year were 
encouraged to attend summer school. Case managers assisted enroUees who 



9 Enrollees who had graduated from high school received some mentoring and assistance in enrolling in 
postsecondaiy education or training between graduation and the end of the fifth year of the demonstration. 

10 0 ur assessment of how well these and other features of the Q 0 P model were implemented in the 
demonstration sites is summarized below and discussed in detail in Maxfield et al. (2003a). 
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were age 16 or older to find summer jobs. Developmental and community 
service activities continued throughout the summer for all enroUees. 

Each of the other three components of the QOP model was geared toward achieving a 
specific program goal. 

• Educational activities were intended to improve academic achievement 
increase the likehhood of completing high school, and increase the hkelihood of 
going on to college or some other postsecondaiy training program. After an 
academic assessment which formed the basis of an individuahzed education 
plan, educational services were to consist of one-on-one tutoring and computer- 
assisted instruction in specific coursework as well as in basic reading and 
mathematics. Educational services also included visiting nearby college 
campuses and other activities designed to promote awareness of and planning 
for college or other postsecondaiy training. 

• Developmental activities included hfe skills and employment-readiness skills 
training that was designed to reduce risky behaviors by improving enroUees' 
decision-making and social skiUs and to prepare enroUees for seeking and 
retaining jobs. D evelopmental activities also promoted cultural awareness and 
provided recreation, which was fun for enroUees and helped them build strong 
relationships with their case managers and peers. 

• Community service activities, such as visiting the residents of a local nursing 
home or volunteering at a local food bank, were designed to help youth develop 
a sense of responsibiUty for the guality of life of others in their neighborhood. 

The QOP model addressed numerous barriers to success by specifying that supportive 
services were to be provided either directly or indirectly through referrals to other resources 
in the community. QOP case managers referred enroUees to community health and mental 
health services; summer jobs programs; and local agencies that provide housing, food, 
income support, or chUd care. 

In addition to supportive services, QOP provided youth with three types of financial 
incentives to attend program activities. The first was a stipend of approximately $1.25 for 
every hour devoted to educational activities, developmental activities that were not purely 
recreational, and community service. A matching amount was either set aside or deposited 
in an accrual account and promised to the enrollee when he or she earned a high school 
diploma or GED certificate and enrolled in coUege, a certified apprenticeship program, an 
accredited vocational/ technical training program, or the armed forces. EnroUees in some 
sites also received bonuses for completing major program activities.^^ 



11 Financial incentives were not provided for time spent being mentored if the mentoring was not part of 
an educational, developmental, or community service activity. 
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QOP also provided financial incentives to program staff. The two Ford-funded sites 
compensated staff entirely through incentive payments based on the time enrollees spent on 
educational, developmental, and community service activities, while some D 0 L-funded sites 
provided bonuses to staff based at least partly on enrollee participation in these program 
activities. 

Although the goals of Q 0 P were similar to those of many other federal youth programs 
or demonstrations— such as Job Corps, Career Academies, the CET demonstration, School- 
to-Work programs. Upward Bound, and T alent Search— Q 0 P's approach to achieving these 
goals, which we have just described, was different. QOP was more intensive and 
comprehensive than most youth programs or demonstrations, and it had a substantially 
greater emphasis on mentoring. QOP also enrolled less motivated youth than most 
programs do because it did not hmit enrollment to those youth who were sufficiently 
motivated to apply to and remain active in the program. QOP explicitly targeted youth with 
lower grades than Upward Bound and T alent Search do, and it included out-of-school youth, 
unlike Career Academies, School- to-Work, Upward Bound, and Talent Search. Unlike Job 
Corps and CET, QOP included in-school youth. Maxfield et al. (2003b) discuss in greater 
deM these and other differences between QOP and other federal youth programs. 

While QOP differed substantially from several other youth programs, it had many 
similarities with WIA youth programs. In contrast to JTPA youth programs, WIA youth 
programs and QOP provide services that are comprehensive and long term, including: 

• Case management and mentoring by a caring adult 

• Tutoring in basic education and study skills as well as close collaboration with 
local high schools and school districts to improve enroUees' educational 
achievement 

• Community service and leadership training 

• Y ear- round services, including a summer jobs program that is integrated into the 
educational component of the program 

• A broad array of supportive services, including transportation, child care, food, 
and emergency financial assistance 
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How Well Was QOP Implemented? 



As we learned from annual site visits, annual QOP conferences, and conference calls 
with QOP staff, two demonstration sites— Fort Worth and Houston— implemented a 
version of QOP that deviated substantially from the program model. The other five sites 
implemented versions that deviated moderately from the model. (See Maxfield et al. (2003a) 
for a detailed description of how the program was implemented in each site.) 

There were two main reasons why the QOP programs implemented by the 
demonstration CBO s did not closely adhere to the Q 0 P model. First, with the exception of 
the Philadelphia site where the program was operated by the CBO that helped to design the 
QOP model and oversaw the previous QOP pilot, local CBO s found implementing Q 0 P to 
be difficult, primarily because QOP was substantially more comprehensive, intensive, and 
complex than their traditional programs. Second, although all sites were encouraged to 
implement aU of the elements of the Q 0 P model, neither DOT nor, to a lesser degree. Ford 
required sites to do so, in part to allow some flexibihty for adjusting implementation to local 
or changing circumstances. 

By some measures, most sites implemented QOP with the prescribed intensity. AH sites 
implemented the prescribed ratio of about 15 to 25 enrollees per case manager. Case 
managers developed deep personal relationships with the 40 to 60 percent of enrollees who 
attended some program activities regularly and addressed a wide range of barriers facing 
those youth. Most case managers stayed with the program for several years, and many 
stayed for the entire five years of the demonstration. QOP's pohcy of providing access to 
services regardless of an enroUee's behavior or status (such as becoming incarcerated, 
moving to another community, or dropping out of high school) was well implemented. 

By other measures, however, the demonstration CBOs did not implement QOP with 
the prescribed intensity. Sites offered fewer than the prescribed number of hours for at least 
one program component, frequently the community service component. Furthermore, the 
demonstration revealed the practical hmitation of QOP's pohcy of case managers being on 
duty or on call for large numbers of hours each week. Such a policy is limited by the case 
managers' personal lives, the physical difficulties of providing services to enrollees who 
moved far away, and the legd hmits on case manager overtime under the Fair Labor 
Standards Act. 

Most sites did not implement the education component effectively. In particular, few 
sites regularly assessed academic performance via achievement tests, no site developed 
individuahzed education plans based on assessment results, no site implemented a sustained 
program of course-based tutoring, and only three sites— Houston, Philadelphia, and 
Yakima— successfully implemented computer-assisted instruction. These hmitations might 
reflect, in part, the fact that QOP case managers were hired based on their training and 
experience in mentoring and dehvering social services rather than teaching, tutoring, or other 
education-related activities. 
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The developmental component was relatively well implemented. Sites offered many 
different activities. Although developmental activities were intended to focus on life skills 
that would enable the youth to avoid risky behaviors, this component included many 
recreational activities at most sites. Nevertheless, participants found recreational activities to 
be fun, and case managers found them to be useful for fostering program participation and 
building strong relationships with and among their enroUees. 

The community service component at most sites did not follow the program model. 
The most common reasons for deviations were the enroUees' lack of interest and the case 
managers' behef that enroUees needed other QOP services more. Most sites decided to 
reallocate their resources away from community service to mentoring, case management, and 
educational activities. 

Most sites operated QOP throughout school-year vacations and the summer months. 
Several sites subsidized the fee for summer school for enroUees who needed it. One site— 
Cleveland— developed its own summer school during a summer in which the local pubUc 
school district did not operate summer school. Case managers reported that many enroUees 
needed both summer school, because of faUing a course during the school year, and a 
summer job, because of being a member of a low- income family. 

EnroUee stipends were well implemented and appeared to be an effective way to attract 
the enroUees to program activities in the first year or two of the demonstration. As enroUees 
aged and could earn much more per hour by working, case managers found that other 
incentives, such as recognition, attention, and prizes, could replace the stipends. 

JTPA accounting regulations prohibited DOL-funded CBOs from estabUshing accrual 
accounts for enroUees. Instead, these CBOs kept informal records of accrual account 
balances and paid those balances to gualifying enroUees at the end of the demonstration. 
According to case managers, the resulting absence of periodic account statements Umited the 
effectiveness of accrual accounts in increasing program participation. Nonetheless, the 
accounts enabled many enroUees to save for postsecondaiy education or training. Account 
balances at the end of the demonstration ranged from a few hundred doUars to nearly 
$10,000, with most being in the range of $1,000 to $3,000. 

Just over two -fifths of the QOP enroUees reported having received the money from 
their accrual accounts approximately two years after the end of the demonstration, when 
most of the enroUees were in their early twenties. The most common uses for the money 
were purchasing supplies for school or a training program (reported by 77 percent) and 
paying tuition (reported by 69 percent). Other common uses were paying for transportation 
or moving expenses (44 percent) and paying for rent or other living expenses (39 percent). 
About 98 percent of the enroUees who received the money from their accrual accounts used 
at least some of the money for one or more of these four purposes. 

Most sites suppUed many of the most commonly needed supportive services, including 
afternoon snacks and transportation to program activities. On the other hand, most sites did 
not meet their enroUees' needs for child care, health and mental health services, substance 
abuse treatment, and famUy counseUng. In fact, QOP proved to be more a prevention 
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program than a remediation program. The most well developed aspects of QOP were 
designed to prevent youth from engaging in risky behaviors. QOP was less well developed 
for providing services to youth facing the consequences of the risky behaviors in which they 
had already engaged. 
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How Much Did QOP Cost? 



The total QOP expenditure per enroUee averaged $25,000 for the full five years of the 
demonstration. The five-year expenditure per enroHee for the DOL-funded sites ranged 
from $18,000 to $22,000.^^ For the two Ford-funded sites, the expenditure per enrollee was 
$23,000 in Y akima and $49,000 in Philadelphia. Compared with the other sites, Philadelphia 
had much higher expenditures per enrollee in aU measured categories: staff wages and 
benefits, student stipends and accrual account contributions, and other costs. 

Annual expenditures at most sites varied over the five years of the demonstration. 
Spending typicdly increased each year during the first four years and decreased during the 
fifth year. QOP coordinators reported that they developed a better understanding of what 
they could do with the money and where they needed to spend it after the first year or two 
of the demonstration. 

These cost figures cover program operations and management, but exclude the cost of 
technical assistance provided by OICA. Because of the anticipated need for technical 
assistance and OICA's experience in both helping to design the QOP model and 
implementing the program in the Philadelphia site for the pilot study. Ford awarded a grant 
to OICA to provide technical assistance for the QOP demonstration. Technical assistance 
included helping sites set up management information software, funding annual week-long 
training conferences for all Q 0 P staff, and answering guestions as needed. 0 ICA provided 
technical assistance for the demonstration at a cost of $1,125,000, or $38,000 per year per 
site (not counting the Philadelphia site itself). In addition to providing technic^ assistance, 
0 ICA operated the Philadelphia site throughout the demonstration. 



12 D 0 L sites were required to match the federal grant with local funds during the first four years of the 
demonstration. However, the Houston site lost its local matching funds during the third and fourth years. 
About one- third of the lost funds were replaced by D 0 L with grant funds received from the 0 ffice of Juvenile 
Justice and D ehnquency Prevention (U.S. D epartment of Justice) for reducing gang activity. D 0 L also allowed 
the value of staff time spent on grant administration to be classified as local matching funds. 
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How Much Did Enrollees Participate in QOP? 



Most QOP enrollees generally did not meet the high participation targets set by 
program developers. According to the Q 0 P participation data for the first four years of the 
demonstration, enrollees spent an average of 177 hours per year on educational, 
developmental, and community service activities— 24 percent of the annual goal of 750 
hours. Enrollees spent an average of 76 hours per year on education (30 percent of the 
goal), 77 hours on developmental activities (31 percent of the goal), and 24 hours on 
community service (10 percent of the goal). The average time spent on these QOP activities 
fell from 247 hours in the first year of the demonstration to 103 hours in the fourth year (see 
Table 1).'^-'^ 

The level of participation varied across enrollees, but few enrollees met the participation 
targets. The percentage of enrollees spending no time at all on QOP activities increased 
steadily from 1 percent in the first year to 26 percent in the fourth year (see Table 1). 
Similarly, the percentage of enrollees who spent more than 325 hours on Q 0 P activities— 
half the target of 750 hours— decreased from 28 percent in the first year to 10 percent in the 
fourth year. Fewer than 5 percent of enrollees met the participation target of 750 hours in 
any year of the demonstration, and in fact fewer than half of all enrollees spent more than 
750 hours on QOP activities during the entire first four years. However, some enrollees 
participated at relatively high levels, with approximately 13 percent spending more than 
1,500 hours on Q 0 P activities during the first four years of the demonstration. 



Table 1. Participation in QOP Activities 



Cumulative Years 
1 through 4 


Year 1 


Year 4 


Average Number of Hours 


708 


247 


103 


Average Hours on Educational Activities 


305 


110 


40 


Average Hours on Developmental Activities 


306 


105 


41 


Average Hours on Service Activities 


97 


32 


22 


No Hours of Participation (percent) 


1 


1 


26 


More Than 100 Hours (percent) 


88 


73 


29 


More Than 375 Hours (percent) 


62 


23 


11 


More Than 750 Hours (percent) 


36 


1 


0 


More Than 1 ,500 Hours (percent) 


13 


0 


0 



Source: QOP Demonstration Management Information System (MIS). 



13 These numbers vary slightly from those in previous reports because of corrections to errors found in 
the participation data. 

1^ Because the Memphis site did not submit participation data for the fourth year of the demonstration, aU 
numbers relating to participation in the fourth year exclude that site. Participation information was received 
from aU other sites for years one through four. 
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Participation also varied substantially from site to site (see Table 2). Participation 
ranged from highs of 345 hours per year per enrollee in the Y aldma site and 244 hours in the 
Philadelphia site to a low of 68 hours in the Fort Worth site. Compared with enroUees at the 
Fort Worth site, enrollees at the Yakima site spent about 7 times as many hours on 
educational activities, 3 times as many hours on developmental activities, 15 times as many 
hours on community service activities, and 5 times as many hours on all three components 
combined. 

The levels of participation might be disappointing for a program based on the belief 
that youth programs must be intensive to be effective.^^ The roughly 12 percent of enroUees 
who spent 100 or fewer hours on QOP activities during the entire demonstration reported 
that they were not interested in those activities or were involved in other after-school 
activities, such as athletics, working, or caring for famUy members. 

However, although the levels of participation did not meet the target set by program 
developers, that target corresponds to a substantial number of hours. In 2000, the average 
number of instructional hours spent in pubhc school by 15-year-old youth was 990 hours 
(U.S. Department of Education 2005; T^le 26-2). Attaining QOP's goal of 750 hours per 
year would have meant that students would have spent the eguivalent of an additional three- 
guarters of a school year on Q 0 P activities. As it was, the average amount of time enrollees 
spent on Q 0 P activities during the first four years— including summers— corresponds to 
^out 72 percent of an extra school year, stUl a substantial investment of time. 

We can also compare participation in QOP with participation in other extracurricular 
activities in which high school students may participate. To meet the target, QOP enroUees 
would have had to spend two hours per day, 365 days per year— 14 hours per week, 52 
weeks per year— in QOP activities. In comparison, in 2002, the average high school 
sophomore spent 4.6 hours per week on extracurricular activities. Students classified as 
"high-intensity participants"— the 25 percent of students who spent the most time on 
extracurricular activities— spent an average of 9 hours per week on extracurricular activities 
(Ingels et al. 2005). Thus, the Q 0 P target level of participation is higher than the amount of 
time most students spend in extracurricular activities during high school. 



Because financial incentives were provided for participation in educational, developmental, and 
community service activities, the participation data do not include time spent being mentored if the mentoring 
was not part of one of those three activities. At least some enroUees might have received substantial 
mentoring. Although the participation data exclude mentoring time, they include for some enroUees bonus 
hours awarded for achieving significant milestones, such as earning a B average or higher during a grading 
period in high school. Such bonus hours could not be distinguished from regular hours spent on educational, 
developmental, and community service activities. 
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Table 2. Participation in QOP Activities by Site 





Overall 


Fort Worth 


Cleveland 


Washington, 

DC 


Average Number of Hours 


Year 1 


247 


120 


285 


177 


Year 4 


103 


23 


49 


77 


Cumulative Years 1 through 4 


708 


273 


603 


611 


Cumulative Level of Participation, Years 1 through 
4 (percent) 


No Hours 


1 


1 


0 


1 


Over too Hours 


88 


84 


85 


89 


Over 375 Hours 


62 


23 


58 


66 


Over 750 Hours 


36 


5 


34 


35 


Over 1 ,500 Hours 


13 


0 


8 


4 




Houston 


Memphis® 


Philadelphia 


Yakima 


Average Number of Hours 


Year 1 


148 


331 


369 


302 


Year 4 


34 


NA 


89 


348 


Cumulative Years 1 through 4 


409 


NA 


975 


1,378 


Cumulative Level of Participation, Years 1 through 
4 (percent) 


No Hours 


1 


0 


2 


2 


Over too Hours 


77 


94 


96 


92 


Over 375 Hours 


51 


75 


74 


84 


Over 750 Hours 


14 


38 


54 


74 


Over 1 ,500 Hours 


1 


11 


24 


42 



Source: QOP Demonstration Management Information System (MIS). 

®Year 4 data not available for Memphis; cumulative amounts scaled to account for that missing year. 
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Estimating the Impactsof QOP 



To estimate the impacts of QOP, we translated each program goal, such as high school 
graduation, into a quantifiable outcome, such as whether a youth graduated from high 
school. We measured each outcome for a group of youth enrolled in QOP and a group of 
statistically identical youth, called the control group. We formed the QOP group and the 
control group at the start of the demonstration by randomly assigning each youth eligible for 
the program to one group or the other. All members of the QOP group were enrolled in 
QOP. Members of the control group were not allowed to participate in QOP and, thus, 
show what would have happened to the enroHees had they not been enrolled. 

We interviewed enrollees and control group members in person in the spring of the 
fourth academic year of the demonstration, that is, just before they were scheduled to 
graduate from high school.^® The survey collected data on risky behaviors and factors that 
assist a youth in resisting negative influences in his or her social environment. At the same 
time, we administered achievement tests in reading and mathematics. During the last— that 
is, the fifth— year of the demonstration, we conducted a telephone survey covering high 
school graduation, postsecondaiy activities, risky behaviors, and (for the enrollee group) 
attitudes toward QOP. Shortly thereafter, we requested transcripts from the high schools 
that sample members had attended since the beginning of the demonstration. Next, we 
conducted a second telephone survey for which interviewing began two years after the end 
of the demonstration when most sample members were 21 or 22 years old (one year after 
the end of the demonstration in the Washington, D C site, when most sample members there 
were 20 or 21 years old). This second telephone survey covered the same broad topics as 
the first telephone survey. 

As the final data collection activity of the evaluation, we conducted a third telephone 
survey a little more than two years after the start of the second telephone survey. 
Interviewing began nearly six years after most sample members were scheduled to graduate 
from high school, at a time when they were 23 to 25 years old (five years after most sample 
members were scheduled to graduate from high school in the Washington, D C site at a time 
when they were 22 to 24 years old). The third telephone survey covered the same topics as 
the second telephone survey, but collected more detailed information about sample 
members' employment activities and earnings. 

The response rates for the in-person survey, the first telephone survey, and the second 
telephone survey were 84 percent, 84 percent, and 75 percent, respectively.^’ For each of 
these surveys, the response rate for the QOP group exceeded the response rate for the 
control group by about 7 to 10 percentage points. The response rate for the third telephone 



16 Exhibit 1 presents key dates pertaining to our data collection activities. 

1’ We collected complete transcript data for 74 percent of sample members and partial academic records 
for another 8 percent of sample members. 
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survey was 76 percent, and the response rate for the Q 0 P group was 3 percentage points 
higher than the response rate for the control group. 



We estimated the impact of QO P on an outcome by subtracting the mean outcome for 
the control group from the mean outcome for the QOP group.^®- For this report, we 
measured outcomes using only data from the third telephone survey, except in the case of 
outcomes pertaining to high school completion. For those outcomes, we used data from the 
first and second telephone surveys and school transcripts in addition to the data collected in 
the third telephone survey, as described in Appendix F.^° 



18 All impact estimates presented in this report are simple difference-of-means estimates, except in 
Appendix F, where we present regression-adjusted impact estimates. Using regression methods allows us to 
adjust for purely random baseline differences between QOP and control group members. With very few 
exceptions, which are noted below, difference-of-means and regression-adjust^ estimates imply the same 
conclusions. 

19 When estimating means for each group, we used weights to adjust for survey nonresponse, as described 
in Appendix E . In Appendix F we assess the sensitivity of our estimates to the models used to derive weights, 
as well to an adjustment for the differential response rate between the Q 0 P group and the control group, and 
find that our estimates are robust, with very few exceptions. In addition to discussing our approach to 
weighting. Appendix E describes in detail how we estimate impacts and their variances, that is, the error 
associated with the impact estimates. 

20 For this evaluation, we are estimating impacts on many different outcomes, including, for example, 
several measures of postsecondaiy education or training activities, several measures of employment, several 
measures of substance abuse, and several measures of criminal activity. We are also estimating impacts for the 
demonstration as a whole, subgroups of sample members, and each demonstration site. When estimating 
impacts for multiple outcomes, there is a concern that some estimated impacts will be found to be significantly 
different from zero, even if there is actually no impact of QOP (a "Type 1" error). In fact, even if there were 
no differences between the QOP and control groups, five percent of estimated impacts would be expected to 
be significant at the five percent level just by chance. Likewise, 10 percent could be significant at the 10 percent 
level just by chance. A variety of procedures have been developed to address the concerns about this. T o 
maintain a straightforward presentation of results, without introducing the complexities of and debate 
surrounding the details of the implementation of multiple comparisons adjustments, we have not included an 
adjustment for multiple comparisons in the tables of results presented in this report. However, we have 
applied two methods that adjust the significance levels of statistical tests to account for the number of tests 
being performed: the Bonferroni correction and a more powerful adjustment developed by Benjamini and 
Hochberg (1995). In particular, when we found significant impacts in the main analyses, we performed the 
multiple comparisons adjustments within outcome groups (for example, outcomes relating to four-year college 
attendance). It is important to note that we generally find few significant impacts in the main analyses, and any 
adjustment for multiple comparisons results in even fewer significant impacts. We discuss our findings from 
these adjustments for multiple comparisons when it appears that one or more significant impacts might be 
attributable to chance rather than the effects of Q Q P. 
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Exhibit 1. Key Dates Pertaining to Data Coiiection for the OOP Evaiuation impact Study 



All Sites except Washington, DC Site 


Students Entered Ninth Grade 


August-September 1995 


In-Person Survey and Achievement Tests 


February-April 1999 


On-Time Graduation Date 


May-June 1999 


First Telephone Survey 


November 1999-June 2000 


School Records Collection 


September 1999-December 2000 


End of Demonstration 


September 2000 


Second Telephone Survey 


September 2002-April 2003 


Third Telephone Survey 


January 2005-September 2005 


Washington, DC Site 


Students Entered Ninth Grade 


August-September 1996 


In-Person Survey and Achievement Tests 


April 2000 


On-Time Graduation Date 


June 2000 


First Telephone Survey 


November 2000-April 2001 


School Records Collection 


December 2000-April 2001 


End of Demonstration 


September 2001 


Second Telephone Survey 


September 2002-April 2003 


Third Telephone Survey 


January 2005-September 2005 
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Impacts ON QOP's First Primary Outcome: High School Completion 



QOP did not significantly increase the likelihood of graduating from high school with a 
diploma (see T able 3). It also did not significantly increase the likelihood of completing high 
school by earning either a diploma or a G ED As discussed in more detail below, the 
lack of an impact on high school completion is consistent with the demonstration sites' 
limited success in implementing the Q 0 P model, and particularly the education component. 

The National Education Longitudinal Study of 1988 (NELS) has followed a cohort of 
students who were eighth graders in 1998 (the 1997- 1998 school year) and provides national 
data on high school completion to which our estimates can be compared. In 2000, eight 
years after scheduled high school graduation, 83 percent of the NELS cohort had earned a 
high school diploma, and 92 percent had earned either a high school diploma or G ED (U.S. 
D epartment of E ducation 2002b). G raduation rates were lower for non-Hispanic blacks and 
Hispanics: 76 percent of non-Hispanic blacks and 75 percent of Hispanics received a high 
school diploma. Rates of completion by earning a diploma or G ED were 90 and 85 percent 
for non-Hispanic blacks and Hispanics, respectively. While these rates are higher than the 
rates we observe for QOP enroHees, completion rates for enroHees are similar to rates for a 
subgroup that had exhibited similarly poor academic performance. Among NELS cohort 



21 Throughout this report, we use the statistical definition of "significant." Under that definition, an 
estimated impact is significant if, according to the available data, it is highly likely that the impact is different 
from zero. That an impact is significant does not imply that it is, for example, large or substantively important. 
When we say in this report that "Q 0 P had an impact" on a particular outcome, that impact is significant unless 
otherwise noted. Likewise, when we say that "Q 0 P did not have an impact," the impact is not significant. 

22 These findings are consistent with the results reported in Schirm and Rodiiguez-Planas (2004), based 
on information collected through the second telephone survey. However, they are inconsistent with the results 
based only on data obtained from the first telephone survey, the in-person survey, and transcripts and reported 
in Maxfieid et al. (2003b). Those earlier results indicated that as of the first telephone survey, QOP increased 
by seven percentage points the likelihood of earning a diploma. Schirm and Rodiiguez-Planas (2004) discuss in 
detail the differences between the estimated short-run impacts and the findings that also use information from 
the second telephone survey. 

23 The sensitivity analyses presented in Appendix F demonstrate that the estimates of Q 0 P's impact on 
the likelihood of graduating from high school are not sensitive to how we adjust via weighting for missing data 
on graduation status; how we measure graduation status using the data now available when, for example, there 
^pear to be inconsistencies between the survey responses of sample members and their previous responses or 
their high school transcripts; or vdiether we use regression methods to adjust for random baseline differences 
between the Q 0 P and control groups. 

24 0 ne potential concern about estimates derived using data from the third telephone survey is that the 
estimated impacts for the Washington, DC, site and, therefore, the estimated impacts for the whole QOP 
demonstration might be affected by the relatively large difference in response rates between QOP and control 
group members in the Washington, DC, site (79 percent versus 67 percent, respectively) and by the fact that 
program operations began a year later and sample members are typically a year younger in the Washington, D C, 
site than in the other six sites. In Appendix F, we assess whether our estimated impacts for the QOP 
demonstration are sensitive to whether we include or exclude the Washington, D C site. We found that they are 
generally not sensitive. 
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members who had scored in the bottom 25 percent on an eighth-grade mathematics 
achievement test, 67 percent received a high school diploma, and 79 percent received a 
diploma or G ED. 



Table 3. Impacts on High School Completion (Percentages) 





QOP Group 


Control Group 




Outcome 


Mean 


Mean 


Impact 


Received HS diploma 


60 


60 


0 


Received HS diploma or GED 


78 


75 


2 



Source: Telephone surveys and transcripts. 



Note: Each impact was derived by subtracting the control group mean from the QOP group mean prior 

to rounding those means; thus, an impact might not equal the difference between the rounded 
means that are displayed. Estimates were obtained using weights to adjust for differences 
between respondents and nonrespondents in baseline characteristics, response rates to the first 
and second telephone surveys, and outcomes measured in the in-person, first, and second 
telephone surveys. The evaluation sample had 580 QOP enrollees and 489 controls. 

* Estimate significantly different from zero at the 90% confidence level, two-tailed test 

** Estimate significantly different from zero at the 95% confidence level, two-tailed test 

*** Estimate significantly different from zero at the 99% confidence level, two-tailed test 
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Impacts ON QOP's Second Primary Outcome: Postsecondary 
Education or Training 



One of QOP's two primaiy objectives was to increase the likelihood that enrollees 
engage in postsecondaiy education or training. However, we find that Q 0 P did not have an 
impact on such postsecondaiy activities (see Table 4). Within five to six years of scheduled 
high school graduation, 61 percent of QOP enrollees and 56 percent of control group 
members had engaged in some type of postsecondaiy education or training, including college 
attendance, vocational or technics school attendance, apprenticeship enrollment, and armed 
forces enlistment.^^ In the same time period, 38 percent of Q 0 P enroUees and 34 percent of 
control group members had enrolled in a two- or four-year college, and 15 to 16 percent of 
both groups had enrolled in a four- year college. Although the percentages of Q 0 P enroUees 
who had enroUed in a four-year college, a two- or four-year college, or engaged in any 
postsecondaiy education or training are higher than the percentages for control group 
members, the differences are not statisticaUy significant.^® 

Although QOP enrollees and control group members engaged in postsecondaiy 
activities at similar rates, it is possible that the QOP enrollees were better prepared to 
undertake and persist in these activities, resulting in higher levels of attainment. However, 
this does not appear to be the case, with QOP enroUees and control group members 
completing postsecondaiy degrees at similar rates. Six percent of QOP enroUees earned a 
bachelor's or associate's degree within about five to six years of scheduled high school 
graduation, as compared with seven percent of control group members, and the difference in 
rates is not statisticaUy significant. QOP also had no impact on the likeUhood of enroUees 
completing a coUege or vocational degree, completing an apprenticeship, being enlisted in 
the mUitaiy for more than two years, or being honorably discharged from the mUitaiy: this 
rate was 22 percent among QOP enroUees and 25 percent among control group members. 



25 A sample member is classified as ever engaging in postsecondaiy education or training if he or she was 
engaged in such activities at the time of the third telephone survey or had previously engaged in such activities. 

26 For comparison with our estimates, national data from NELS indicate that 76 percent of eighth graders 
in 1988 enrolled in postsecondaiy education (coUege, university, or vocational or technical school) within eight 
years of scheduled high school graduation (U.S. D epartment of E ducation 2002b). The rates for non-Hispanic 
blacks and Hispanics are 76 percent and 70 percent, respectively. These numbers are slightly higher than the 
64 percent of QOP enrollees we observe as having been engaged in postsecondaiy activities. However, the 
sample members in this study had relatively low eighth-grade GPAs, and the rates of postsecondaiy 
engagement among the QOP enrollees are more similar to the rates for students who had similarly low 
academic performance: 58 percent of eighth- graders with mathematics achievement test scores in the bottom 
25 percent enrolled in postsecondaiy education at some point within eight years of scheduled high school 
graduation (U.S. Departinent of Education 2002b). A similar pattern of differences is seen when examining 
current postsecondaiy activities. N ational data for 2004 show^ that 26 percent of individuals age 22-24 (23 
percent of non-Hispanic blacks and 18 percent of Hispanics) were currently enrolled in college 
(www.census.gov, accessed January 4, 2006), which is somewhat higher than the 13 percent of QQP enrollees 
enrolled m college at the time of the thud telephone survey. 
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These postsecondaiy completion rates are likely to increase at least somewhat over time 
given that about 8 percent of sample members were enrolled in a four-year college at the 
time of the third telephone survey and about 23 percent were engaged in some form of 
postsecondaiy education or training. National data show that over 45 percent of 1999-2000 
bachelor's degree recipients completed their degree more than five years after high school 
graduation (U.S. Department of Education 2003). Although no further data collection 
activities are planned, it is possible that either a beneficial or detrimental impact on the 
attainment of postsecondaiy education or training could emerge in the future. 

As demonstrated in Appendix F, most of our findings pertaining to postsecondaiy 
education or training activities are not sensitive to whether we use regression methods to 
adjust for random basehne differences between the QOP and control groups. One 
exception is that, according to regression-adjusted impacts and impacts estimated using 
some of our other methods of assessing the robustness of our findings, QOP's impact on 
engagement in any postsecondaiy education or training is statistically significant according to 
our least- stringent criterion for assessing significance (see Appendix F).^’ However, when 
we adjust the significance levels of the regression-adjusted estimates to account for the many 
outcomes that are considered, we find that none of the impacts on postsecondaiy education 
or training are statistically significant. Thus, we do not find consistent evidence of an impact 
of Q 0 P on the program's second piimaiy outcome. 

These findings differ from the beneficial impacts on postsecondaiy education and 
training estimated from data obtained in the first and second telephone surveys (Maxfield et 
al. 2003b and Schirm and Rodriguez- Planas 2004). In particular, Maxfield et al. (2003b) 
reported that QOP enroUees were more likely than control group members to be engaged in 
postsecondaiy education or training or to have been accepted by a college within about one 
year of scheduled high school graduation. Similarly, Schirm and Rodriguez- Planas (2004) 
found that QOP enroUees were more hkely to enroll in college or engage in any 
postsecondaiy education or training within about three years of scheduled high school 
graduation. 

This departure from the findings from previous reports, with no impacts on 
postsecondaiy attainment using data from the thinl telephone survey, seems to be primarily 
due to a larger number of control group members than QOP enroUees engaging in 
postsecondaiy education in the two years between the second and third telephone surveys. 
While the percentage of QOP enroUees who were ever engaged in any postsecondaiy 
education or training increased by only one percentage point (to 62 percent) during this time, 
the control group percentage increased by three percentage points (to 56 percent). SimUar 
differences are seen for four-year college enroUment and two- and four-year college 
enrollment, with the gap between the Q 0 P enroUees and the control group narrowing by 2 
to 3 percentage points between the two surveys. The changes seen over time are perhaps 
not surprising given that many students do not engage in postsecondaiy education directly 
after high school. For example, over 10 percent of 1999-2000 bachelor's degree recipients 



2’ The p-value on the regression-adjusted impact for engagement in any postsecondaiy education or 
training is 0.06. The p-value on the corresponding difference-of-means estimate is 0.14. 
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enrolled in college more than 2 years after high school graduation (U.S. Department of 
Education 2003). 
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Table 4. Impacts on Postsecondary Attainment (Percentages) 
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Estimate significantly different from zero at the 90% confidence level, two-tailed test 
Estimate significantly different from zero at the 95% confidence level, two-tailed test 
Estimate significantly different from zero at the 99% confidence level, two-tailed test 




Impacts ON Employment and Earnings 



Improving enrollees' prospects for having successful careers was a principal motivation 
for QOP's two primaiy objectives of increasing the rates of high school graduation and 
engagement in postsecondaiy education or training. Thus, improved employment outcomes 
were an imphcit, long-run goal of the program. Although the final data available from the 
evaluation were collected when sample members were approaching their mid-twenties, we 
can assess whether QOP increased the hkelihood of being employed and the amount earned 
early in enroUees' working hves. 

QOP did not affect the hkelihood of employment (see Table 5). At the time of the 
third telephone survey, about two-thirds of both QOP enrohees and control group members 
were employed, with just over half of each group working a fuh-time job (at least 35 hours 
per week). When we consider the employment experiences of sample members during the 
year preceding the survey, we do not find significant differences between QOP enrohees and 
control group members in the hkelihood of employment or the fraction of time 
employed.^*^'^'^ 

Our findings of no impacts on the hkelihood of employment or full-time employment 
at the time of the third telephone survey differ from our previous findings (Schirm and 
Rodriguez-Planas 2004). At the time of the second telephone survey two years earlier, 
control group members were more hkely than QOP enrollees to have a job or a fuh-time 
job. The differences in findings stem from both an increase in the likelihood of employment 
or fuh-time employment among QOP enrollees and a decrease in the hkelihood among 
control group members. 

When we examine the compensation associated with jobs held by sample members, we 
find that QOP did not increase the earnings of QOP enrollees or the avail^hity of benefits 
in the jobs held by enrohees. T otal earnings in the year preceding the survey and hourly 
earnings at the time of the survey are higher among control group members than among 
QOP enrohees, but the differences are not statisticahy significant. Similarly, the percentages 



28 As noted in Table 3, when we examine a broader measure of postsecondaiy activity that counts both 
employment and education or training, we find that Q 0 P did not affect the hkelihood of being employed or 
engag^ in postsecondaiy education or training at the time of the third telephone survey. This lack of an 
impact is not surprising given that we do not find impacts on either employment or engagement in 
postsecondaiy education or training. About one-quarter of sample members were neither employed nor 
engaged in postsecondaiy education or training. The most commonly given reasons for not worMng among 
both QOP enrollees and control group members were that the sample member was looking for work or had to 
stay home with children. 

29 According to Current Population Survey data for the civihan noninstitutional population 
(www.bls.gov/cps/home.htm, accessed Januaiy 11, 2006), 56 percent of blacks and 68 percent of Hispanics 
age 20 to 24 were employed in 2004, while 40 percent and 51 percent were employed full-time. Thus, the rates 
of employment and full-time employment among QOP enroUees are roughly similar to— and perhaps higher 
than— the rates among young adults of similar race and ethnicity; however, because most QOP enrollees were 
in the upper portion of the 20 to 24 age range, we might expect their employment rates to be higher. 
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of control group members with jobs that offer health insurance, paid time off, or pension or 
retirement benefits are not significantly different from the percentages of Q 0 P enrollees 
with jobs offering such benefits. 

Our finding of no impacts on earnings is consistent with our finding of no impacts on 
high school completion and postsecondaiy attainment. Q 0 P sought to prepare enrollees for 
good jobs by helping them to graduate from high school and obtain postsecondaiy 
education or training. Individuals who delay employment to obtain further education or 
training may initially have lower earnings upon entering the labor force than similarly aged 
but less educated individuals who have accrued more work experience.^® Had QOP 
enroUees achieved significantly higher rates of high school graduation and postsecondaiy 
education or training than control group members, the absence of a significant increase in 
the earnings of Q 0 P enrollees might be attributable to Q 0 P enrollees experiencing such a 
temporary period of relatively low earnings before their earnings surpass the earnings of less 
educated workers. However, since we do not find significant impacts on high school 
graduation and postsecondaiy education or training, the finding of no impact on earnings 
does not appear to be attributable to this effect of initially reduced earnings due to loss of 
work experience. 



30 Jacob Mincer (1974) introduced this concept in his early work on human capital. 
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Table 5. Impacts on Employment and Earnings (Percentages, Unless Stated Otherwise) 
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Estimate significantly different from zero at the 90% confidence level, two-tailed test 

Estimate significantly different from zero at the 95% confidence level, two-tailed test 

Estimate significantly different from zero at the 99%> confidence level, two-tailed test 




Impacts ON QOP's Secondary Outcomes: High School Performance 

AND Risky Behaviors 



QOP's secondaiy objectives were to improve enrollees' academic performance while in 
high school and reduce their engagement in risky behaviors. As discussed in Maxfield et al. 
(2003b), Q 0 P did not improve achievement test scores, grades, or credits earned in high 
school, and it did not reduce disciplinaiy actions (see T able 6).^^ 

With respect to risky behaviors, Maxfield et al. (2003b) found that when most enroUees 
were in their late teens, Q 0 P did not reduce any risky behaviors, and according to data from 
the in-person survey, it increased some risky behaviors, specifically the fraction of enrollees 
who had a drink and the fraction of enroUees who used an Ulegal drug in the 30 days before 
the survey (see Table 1 )?'^ The only indication of a beneficial impact of QOP on risky 
behaviors was that, at a time when most sample members were in their early twenties, QOP 
enrollees were less likely to use iUegal drugs in the 30 days before the survey. However, 
results based on the same survey found that QOP did not reduce the hkehhood of binge 
drinking, committing a crime, being arrested or charged with a crime, or having a chUd 
before the age of 18 (Schirm and Rodriguez- Planas 2004). These findings pertained when 
most sample members were in their early twenties and, in particular, over the age of 21. 
What were QOP's impacts on risky behaviors as sample members entered their mid- 
twenties? 

We find that QOP did not reduce risky behaviors when QOP enrollees were entering 
their mid-twenties, and in fact increased criminal activities and arrests among enrollees 
during that time, although the evidence for these detrimental impacts is not consistent (see 
Table 8).^^ In particular, QOP did not decrease tobacco use, binge drinking, or iUegal drug 
use, and it increased the percentage of sample members who committed a crime in the three 
months before the survey (from 2 percent to 5 percent) and the percentage who were 
arrested or charged with a crime in the prior two years (from 5 to 11 percent).^^ Although 



31 We have not administered another round of achievement tests or collected additional transcript data, 
and, in the second and third telephone surveys, we did not attempt to obtain further information about 
academic performance while in high school, except to ascertain graduation status. 

32 By paying stipends and bonuses for participation, QOP might have provided some enrollees with the 
money to buy alcohol and drugs. By bringing enrollees together through program activities, QOP might have 
introduced some negative peer effects and facilitated the spread of drinking and drug use. It is also possible— 
and may be likely— that the detrimental effects were not caused by QOP, as discussed in detail in 
Maxfield et al. (2003b) and Schirm et al. (2003). As reported in the latter, data collected in the first telephone 
survey reveal that QOP had beneficial— but not significant— impacts on drinking and drug use. 

33 All information on risky behaviors based on sample member self-report. 

34 Across aU sites, the mean rates for binge drinking and illegal drug use are similar to the national rates 
for non-Hispanic blacks and Hispanics age 18 to 25. According to 2002 data from the National Survey on 
Drug Use and Health, 26 percent of non-Hispanic blacks and 35 percent of Hispanics engaged in binge 
drinking in the 30 days before the survey, while 18 and 14 percent used an illegal drug (www.oas.samhsa.gov, 

(rontinued) 
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we also find an increase in the percentage committing a crime in the two years before the 
survey, the impact is not statistically significant.^^-^® 

The reason that Q 0 P sought to reduce risky behaviors was that developers of the Q 0 P 
model shared the widespread belief that engagement in risky behaviors creates barriers to 
high school graduation, postsecondaiy education and training, and productive careers. QOP 
did not achieve decreases in targeted risky behaviors— substance abuse, crime, and teen 
parenting— or an increase in high school graduation or postsecondaiy education and 
training. However, it is possible that the case management, mentoring, and developmental 
activities that were undertaken to reduce risky behaviors might have improved the family 
lives of enrollees by, for example, fostering the attitudes and interpersonal skills that 
promote better relationships mth spouses, significant others, and children. We have found, 
however, that Q 0 P has not reduced the likelihood that an enrollee is a single parent or that 
an enroUee has a child with whom he or she is not living (see Table 8). We have also found 
that QOP has not decreased the likelihood of living in a household that receives public 
assistance or of being in poor health.^’ The lack of beneficial impacts on family life and the 
lack of beneficial impacts on risky behaviors is consistent mth the lack of an impact on 
earnings, which could provide financial support for a family, and the lack of an impact on 
the likelihood of being either employed or engaged in postsecondaiy education or training, 
activities that could divert a young adult from engaging in risky behaviors. 



(cnntinued) 

accessed May 21, 2004). (As noted above, about two-thirds of the members of the QOP demonstration 
sample are non-Hispanic black, and just over one-quarter are Hispanic.) 

35 When we excluded the Washington, DC site, where sample members are typically one year younger 
than in the other sites, we find that QOP also increased the proportion of Q 0 P enrollees who were frequent 
binge drinkers (8 or more days in the past month) and the proportion of enrollees who committed a crime in 
the past two years. When the significance levels are adjusted to account for the multiple hypothesis tests being 
performed, the impact on binge drinking and one of the impacts on criminal activity (arrested or charged in the 
past two years) remain significant. 

36 As demonstrated in Appendix F, our findings pertaining to risky behaviors, physical and mental well- 
being, and family life are generally not sensitive to whether we use regression methods to adjust for random 
baseline differences between the QQP and control groups. The exceptions are that the regression-adjusted 
impacts on whether individuals committed a crime in the past three months and whether individuals had been 
arrested or charged in the past two years are not statistically significant, whereas the difference-of-means 
estimates are significant. 

33 When evaluating QQP's effectiveness, it is important to understand that improving the quality of an 
enroUee's family life in young adulthood along these dimensions was not a stated objective of the program. 
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Table 6. Selected Impacts on High School Performance 



Outcome® 


QOP Group 
Mean 


Control Group 
Mean 


Impact 


Mathematics achievement test score (percentile) 


40.9 


40.5 


0.4 


Reading achievement test score (percentile) 


43.2 


42.7 


0.5 


Cumulative GPA (four-point scale) 


2.13 


2.19 


-0.06 


Total credits (Carnegie units) 


16.2 


15.8 


0.5 


Suspended or expelled in past 12 months (percentage) 


34 


38 


-4 



Source: Maxfield et al. (2003b), Table 3 

Note: Each impact was derived by subtracting the control group mean from the QOP group mean prior 

to rounding those means; thus, an impact might not equal the difference between the rounded 
means that are displayed. Estimates were obtained using weights to adjust for differences 
between respondents and nonrespondents. The evaluation sample had 580 QOP enrollees and 
489 controls. 

® Achievement test scores are expressed as percentiles in the distribution of scores for tenth graders in the 
United States. Credits are expressed in Carnegie units that standardize for in-class time. One 
Carnegie unit corresponds to a class that meets for 45 to 60 minutes every day of the week for an 
entire academic year. 

* Estimate significantly different from zero at the 90% confidence level, two-tailed test 

** Estimate significantly different from zero at the 95% confidence level, two-tailed test 

*** Estimate significantly different from zero at the 99% confidence level, two-tailed test 
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Table 7. Selected Impacts on Risky Behaviors When Enrollees Were In Their Late Teens and Early- 
Twenties (Percentages) 

QOP Group Control Group 



Outcome 


Mean 


Mean 


Impact 


When Enrollees Were in Their Late Teens® 


Drinking in the past month 


40 


33 


y** 


Binge drinking in past month 


24 


20 


4 


Used an illegal drug in the past month 


34 


28 


y** 


Committed a crime in the past 12 months 


31 


28 


3 


Ever pregnant or get anyone pregnant 


33 


33 


0 


Have had a child 


23 


26 


-3 


When Enrollees Were in Their Early Twenties'^ 


Binge drinking in past month 


25 


31 


-6 


Used an illegal drug in past month 


12 


18 


-6** 


Committed a crime in past 3 months 


8 


9 


-2 


Had first child before age 18 


19 


15 


3 


Currently receiving welfare or food stamps 


24 


20 


4 



Sources: ® Maxfield et al. (2003b), Table 4 

“^Schlrm and Rodriguez-Planas (2004), Table 4 



Note: Each Impact was derived by subtracting the control group mean from the QOP group mean prior 

to rounding those means; thus, an impact might not equal the difference between the rounded 
means that are displayed. Estimates were obtained using weights to adjust for differences 
between respondents and nonrespondents. The evaluation sample had 580 QOP enrollees and 
489 controls. 

* Estimate significantly different from zero at the 90% confidence level, two-tailed test 

** Estimate significantly different from zero at the 95% confidence level, two-tailed test 

*** Estimate significantly different from zero at the 99% confidence level, two-tailed test 
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Impacts ON Subgroups 



Impact estimates for the full evaluation sample might conceal important differences in 
impacts across subgroups. If an impact exists overall, it might be heavily concentrated in or 
could be much larger for some subgroups. Conversely, if an impact does not exist for the 
entire QOP target group, it might still exist for some subgroups. Thus, estimates of 
subgroup impacts can help pohcymakers identify the persons for whom a program is most 
effective and thereby better target a program or better tailor its services. 

We present impacts for subgroups defined by baseline characteristics— sex, age, and 
38,39 examining subgroup impacts, we present impacts for each of the seven 
demonstration sites. 

The QOP demonstration was designed primarily to estimate demonstration- wide 
impacts. Thus, the sample for a subgroup or individual site is small, generally reducing the 
precision of impact estimates and making it difficult to be confident that an estimated impact 
is significantly different from zero.^® 

All of the tables of subgroup and site impacts present two types of significance tests. 
One test is whether the impact is significantly different from zero, as indicated by asterisks. 
The other test is whether the impact for one subgroup is different from the impact for all of 
the other subgroups combined, as indicated by daggers (t). The conclusions presented in 
the text are based on whether the impacts are significantly different from zero, unless 
otherwise noted. 



38 The subgroups examined are overlapping and not mutually exclusive because the baseline 
characteristics defining them are related. For example, males are more likely to be in the bottom third of the 
baseline grade distribution than are females. Furthermore, compared with sample members who were age 14 
or younger when they entered ninth grade, those who were over age 14 are more likely to be male and in the 
bottom third of the grade distribution. G iven such relationships, an impact on the older sample members, for 
instance, might be attributable to the effects associated with being older, being male, or having lower grades. 
Although such effects could potentially be disentangled by defining subgroups based on two (or three) baseline 
characteristics— rather than just one characteristic— sample sizes are too small to allow us to obtain impact 
estimates that are sufficiently reliable to be informative. We note also that we do not find consistent patterns 
of impacts across subgroups that have substantial overlap, suggesting that such overlap does not help to 
explain the findings. This overlap also raises the question of what characteristics of enroUees are most 
important for identifying enroUees who wiU benefit from QOP. There is no way to obtain a definitive answer 
to this question, but some exploratory analyses suggest that an enrollee's age— whether the enroUee was over 
14 when entering ninth grade— is the primary characteristic that determines whether the enrollee wiU benefit 
from QOP. 

39 To adjust for random differences that may exist between the treatment and control groups within each 
subgroup, we also obtained regression-adjusted subgroup estimates, using the method described in Appendix 
F. We generaUy found results very similar to those presented here; any differences are discussed below. 

49 Site and subgroup sample sizes are reported in Appendices C and D (Tables C.3 and D .2). 
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Impacts by Sex 

Q 0 P registered some impacts for both males and females (see T ables 9-12). However, 
Q 0 P does not seem to have consistently benefited one group more than the other.^^ 

For females, QOP had detrimental impacts on engagement in postsecondaiy education 
or training at the time of the third telephone survey, but beneficial impacts on the positive 
attributes of jobs held.^^-^^-^^ QOP decreased by 6 percentage points the Ukehhood of being 
enrolled in a four-year college and by 11 percentage points the hkehhood of being enrolled in 
either a two- or a four-year college. It dso decreased by 8 percentage points the hkehhood 
of females being engaged in any postsecondaiy education or training. However, QOP 
increased the likelihood of females having a job with health insurance (by 10 percentage 
points) or pension and retirement benefits (by 9 percentage points).^^ These impacts were 
significantly different from the impacts on males. 

For males, QOP had one beneficial impact: a 7-percentage-point increase in the 
hkehhood of receiving a high school diploma or G E D , although the evidence for this impact 
is not consistent.^® It also had some detrimental impacts. In contrast to our finding for 
females, we find a decrease in the likelihood of males having a job with positive attributes: 
QOP decreased by 15 percentage points the hkehhood of males having a job with health 
insurance, by 11 percentage points the likelihood of having a job with paid time off, and by 
11 percentage points the likehhood of having a job with pension or retirement benefits. 
QOP also possibly increased the hkehhood of criminal activity among males. The likehhood 
that QOP enrohees committed a crime in the three months before the survey increased by 5 
percentage points. When we consider activity in the two years before the survey, we find 
that QOP increased the hkehhood of committing a crime by 10 percentage points, the 
hkehhood of being arrested or charged with a crime by 12 percentage points, and the 
hkehhood of being convicted of or pleading guilty to a crime by 4 percentage points. Some 
of these effects on criminal activity by males are sensitive to alternative estimation methods. 



About half of Q 0 P enroUees were male. 

42 We do not find evidence that QOP reduced postsecondaiy attainment for females. However, the lower 
rates of engagement among female QOP enrollees relative to female control group members at the time of the 
survey suggest that a detrimental impact on postsecondaiy attainment might emerge. 

43 We also find a beneficial impact for females on the likelihood of committing a crime in the two yeans 
before the survey. However, when we adjust significance levels for the multiple criminal activity outcomes that 
are considered, we find that the impact is not statistically significant. 

44 When we adjust for random baseline differences using regression methods, we find that QQP 
significantly decreased the percentage unemployed for females, by 6 percentage points, and also decreased 
criminal activity among females. 

43 This impact on having a job with retirement benefits is somewhat sensitive to regression- adjustment; 
the regression-adjusted impact of 9 percentage points is not statistically significant (p-value=0.11). 

46 The impact on the likelihood that a male enroUee earned a high school diploma or G E D has a p-value 
of 0.099, and is, thus, just barely significant at the 0.10 level, our least stringent criterion. When we examine the 
alternative measures of high school completion discussed in Appendix F, we find that the impacts are not 
statistically significant. 
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but others are robust and we thus find strong evidence that QOP increased at least some 
types of criminal activity among males.^’ 

How do these findings compare with the impacts estimated previously from the second 
telephone survey (Schirm and Rodriguez- Planas 2004)? As we observed for the sample as a 
whole, we no longer find beneficial impacts among males or females on postsecondaiy 
education and training.^*^ Furthermore, while QOP reduced substance abuse by males when 
they were in their early twenties, we do not find this impact persisting as they enter their 
mid-twenties.^® Finally, previously unobserved detrimental impacts have emerged on 
postsecondaiy engagement for femmes, on the availability of job benefits for males, and on 
criminal activity for males. 



When we adjust significance levels for multiple comparisons using the Bonferroni correction or when 
we adjust for random baseline differences using regression methods, only the impact on the likelihood of being 
arrested or charged with a crime in the past two years remains statistically significant. However, when we use 
the more powerful adjustment developed by Benjamini and Hochberg, aU four of these impacts on criminal 
activity remain statistically significant, providing strong evidence of an impact on criminal activity for males. 

We previously found that QOP increased by 9 percentage points the likelihood of males ever attending 
college and by 10 percentage points the likelihood of females ever engaging in any postsecondaiy education or 
training. 

® We also found in our previous analysis a detrimental impact on frequent binge drinking among females. 
However, as discussed in Schirm and Rodriguez- Planas (2004), QOP may not have caused the estimated 
impact. In our current analysis, we find no impact on frequent binge drinking. 
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Table 9 . Impacts on High School Completion by Sex (Percentage Points) 
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Table 10. Impacts on Postsecondary Attainment by Sex (Percentage Points) 
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Table 11. Impacts on Employment and Earnings by Sex (Percentage Points, Unless Otherwise Indicated) 
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Estimate significantly different from zero at the 90% confidence level, two-tailed test 

Estimate significantly different from zero at the 95% confidence level, two-tailed test 

Estimate significantly different from zero at the 99% confidence level, two-tailed test 
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Estimate significantly different from zero at the 90% confidence level, two-tailed test 
Estimate significantly different from zero at the 95% confidence level, two-tailed test 
Estimate significantly different from zero at the 99% confidence level, two-tailed test 




Impacts by Age When Entering N inth Grade 

About two-thirds of all QOP enroUees were age 14 or younger when they entered the 
ninth grade. QOP increased rates of high school completion and engagement in 
postsecondaiy education and training for these younger enrollees but not for the older 
enrollees who were over age 14 when they entered the ninth grade. We do not find a 
consistent pattern of impacts on other categories of outcomes (see Tables 13-16). 

For younger enrollees, QOP increased by 7 percentage points the hkelihood of 
receiving a diploma and by 6 percentage points the hkehhood of receiving a diploma or 
GED (see Table 13).'’°'’^ The first of these two beneficial impacts is significantly different 
from the impact on older enroUees.^ 

QOP's beneficial impact on younger enrollees' postsecondaiy attainment also contrasts 
with the lack of impact on older enrollees' postsecondaiy attainment. For younger enroUees, 
QOP increased by 10 percentage points both the likelihood of ever attending postsecondaiy 
education or training and the likelihood of completing at least two years of coUege or 
mUitaiy service or completing vocational/ technical school or an apprenticeship (see Table 
14). There were no impacts on postsecondaiy attainment for older enrollees. 

There are several employment and earnings outcomes for which the impacts for older 
and younger enroUees are significantly different from each other (see Table 15). However, 
there are only two impacts that are also significantly different from zero: the impact on the 
number of jobs in the year before the survey for younger enroUees (an increase of 0.2 jobs) 
and the impact on tenure at current job for older enroUees (a decrease of 6 months). The 
interpretation of these impacts is ambiguous. Having more jobs in the past year or a shorter 
tenure at the current job is a poor outcome if it reflects an inabUity to maintain employment. 
However, such an outcome is a good outcome if the enroUee left a job with poor attributes 
to begin a job with better attributes. 

We find a few detrimental impacts on risky behaviors and famUy life for both younger 
and older enroUees (see Table 16).'’^ However, when we adjust significance levels for 



50 When we examine alternative measures of high school completion, the impacts on younger enrollees 
remain significant for aU but the measure based only on sample members who responded to the third survey 
(the "First Alternative" described in Appendix F). 

51 There is also some evidence that QOP may have decreased the percentage unemployed among younger 
enroUees: the regression-adjusted impact is -5 percentage points and statistically significant at the 90 percent 
confidence level. 

52 Although we do not find impacts on high school completion or other outcomes for the population of 
all QOP enrollees, it is still possible that some enroUees did benefit from QOP services. For example, QOP 
may have been more effective for younger enroUees, who had not experienced such severe academic difficulties 
as having faUed a grade prior to high school, which was much more common among older enroUees. 

53 Among younger enroUees, QOP appears to have increased the likeUhood of being arrested or charged 
and the likeUhood of being convicted of or pleading guUty to a crime in the two years before the survey. QOP 
also increased the likelihood of older enroUees being arrested or charged and the likelihood of having children 

(rontinued) 
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multiple comparisons or obtain regression-adjusted impacts (see Appendix F), these impacts 
are no longer statistically significant. We thus do not find strong evidence of impacts on 
risky behaviors or family life for either group. 

Comparing the impacts estimated from the third telephone survey and reported here 
with the impacts estimated from the second telephone survey and reported in Schirm and 
Rodriquez-Planas (2004), we find that the pattern across age groups is generally similar for 
impacts on postsecondaiy attainment, although we no longer find impacts on college 
attendance and completion for younger enroUees. Nonetheless, as before, we find more 
favorable impacts on postsecondaiy education or training for younger enrollees than for 
older enroUees. In contrast, we find somewhat different patterns of impacts on employment 
and on risky behaviors and famUy life. We no longer find detrimental impacts for younger 
enroUees on employment or fuU-time employment. Finally, while we found no impacts on 
high school completion before, we now find that Q 0 P increased high school completion 
rates among younger enroUees. 



(rontinued) 

with whom they are not currently hving. For only the last of these outcomes is the impact on older enrollees 
significantly different from the impact on younger enrollees. 
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Table 13. Impacts on High School Completion by Age When Entering Ninth Grade (Percentage Points) 
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Table 14. Impacts on Postsecondary Attainment by Age When Entering Ninth Grade (Percentage Points) 
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Table 15. Impacts on Employment and Earnings by Age When Entering Ninth Grade (Percentage Points, Unless Otherwise Indicated) 

Impacts 

Outcome Age > 14 Age < 14 Total Sample 

Currently employed -4 3 -1 

Currently unemployed 2 -6 -1 
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Estimate significantly different from zero at the 90% confidence level, two-tailed test 
Estimate significantly different from zero at the 95% confidence level, two-tailed test 
Estimate significantly different from zero at the 99% confidence level, two-tailed test 




Impacts by Rank in the Baseline Grade Distribution 

When assessing impacts for the subgroups defined by rank in the baseline (eighth- grade) 
grade distribution, it is important to remember that, to be eligible for QOP, a youth had to 
be in the bottom two -thirds of the grade distribution based on grades from the eighth grade. 
We defined the subgroups by dividing each QOP school's evaluation sample into thirds. 
Thus, for example, the youth in the middle third of the evaluation sample feU between 
roughly the 22nd and 44th percentiles in the grade distribution for aU entering ninth graders. 

Across the three subgroups of enrollees defined by rank in the baseline grade 
distribution, we find few significant impacts and, thus, no strong, consistent patterns of 
impacts (see Tables 17-20). For enroUees in the bottom third of the distribution, QOP 
increased by 12 percentage points the hkelihood of completing at least two years of college 
or mUitaiy service or completing vocational/ technical school or an apprenticeship. We also 
find that QOP decreased by 7 percentage points the hkehhood of being enrolled in college at 
the time of the third telephone survey. However, this impact on college enrollment was not 
statistically significant when we adjusted significance levels for the multiple outcomes 
considered. QOP had no other impacts on enrollees in the bottom third of the baseline 
grade distribution. Overall, we find only weak evidence of beneficial impacts on enroUees in 
the bottom third of the basehne grade distribution. 

For enroUees in the middle third of the baseline grade distribution, QOP had no 
beneficial impacts and detrimental impacts on some outcomes pertaining to criminal 
activity.^^ It increased by 9 percentage points the hkehhood of committing a crime in the 
three months before the third telephone survey. QOP also increased the hkelihood of being 
arrested or charged (by 7 percentage points), being convicted of or pleading guhty to a crime 
(by 4 percentage points), and serving time (by 4 percentage points) in the two years before 
the survey. However, only the impact on the hkehhood of having committed a crime in the 
past three months was significantly different from the impacts on ah other enroUees (see 
Table 20). Thus we find strong evidence of a detrimental impact on criminal activity in the 
three months prior to the survey but weaker evidence of detrimental impacts on 
involvement with the criminal justice system in the two years before the survey. 

Finally, for enroUees in the top third of the basehne grade distribution, QOP had one 
detrimental impact but no beneficial impacts.'’^' QOP increased by 9 percentage points the 



54 When we adjust for random baseline differences using regression methods, we find a statistically 
significant increase of nine percentage points in the completion of a high school diploma or G E D among QOP 
enrollees in the middle third of the grade distribution (p- valuer 0.04). The non-regression-adjusted impact is 
seven percentage points and not statistically significant (p-value=0.23). 

55 There is some evidence of an increase in postsecondary activities and employment among enrollees in 
the top third of the grade distribution. When we adjust for random baseline differences using regression 
methods, we find that QOP increased by 10 percentage points the percentage of these enrollees who are 
currently out of the labor force (p-value=0.09). The non-regression-adjusted impact is 9 percentage points but 
not statistically significant (p-value=0.11). The regression-adjusted impact on the percentage ever employed is 
also statistically significant (an increase of 3 percentage points, p-value=0.07), while the non-regression- 
adjusted impact is of the same magnitude but not statisticdly significant (p-value=0.32). In both cases it is 

(rontinued) 
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likelihood of living in a household receiving welfare. However, the evidence of an impact is 
weak because this impact is not significant when we adjust significance levels for multiple 
comparisons. 

In our previous analyses of data collected when most sample members were in their late 
teens and data collected when most sample members were in their early twenties, we found, 
respectively, that Q 0 P seemed to be most beneficial for enroUees in the middle third of the 
baseline grade distribution (Maxfield et al. 2003b) and for enroUees in the bottom two-thirds 
of the basehne grade distribution (Schirm and Rodriguez-Planas 2004).^® In the current 
analysis of data coUected when most sample members were entering their mid-twenties, 
however, we have not found convincing evidence of any such patterns.®’ 



(continued) 

likely that the added precision gained through regression-adjustment decreases the variability in the estimates 
and thus leads to significant estimates. 

56 For enroUees in the bottom third of the distribution, Schirm and Rodriguez-Planas (2004) found that 
QOP increased the likelihood of ever attending a four-year coUege by 7 percentage points and of attending 
either a two- or a four-year college by 14 percentage points, while decreasing the likelihood of iUegal drug use 
by 10 percentage points. They found for enroUees in the middle third of the distribution that QOP increased 
by 7 percentage points the likelihood of completing at least two years of coUege. It also decreased enroUees' 
Ukelihood of iUegal drug use (by 8 percentage points), their UkeUhood of committing a crime (by 6 percentage 
points), and their Ukelihood of having chUdren with whom they were not Uving at the time of the second 
telephone survey (by 8 percentage points). 

57 0 ur current findings do not imply, however, that when sample members were younger, Q 0 P did not 
have the beneficial Impacts on risky behaviors described hi the previous note. 
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Table 17. Impacts on High School Completion by Rank in the Baseline Grade Distribution (Percentage Points) 
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Table 18. Impacts on Postsecondary Attainment by Rank in the Baseline Grade Distribution (Percentage Points) 
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Table 19. Impacts on Employment and Earnings by Rank in the Baseline Grade Distribution (Percentage Points, Uniess Otherwise indicated) 
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Impacts BY Site 



Examining post-intervention impacts derived from data collected in the third telephone 
survey, we find that impacts appear to vaiy vridely by site (see Tables 21-24). However, 
many seemingly large differences are not significant because of small sample sizes. 
Moreover, whether we consider all of the estimates or just the significant estimates, the 
patterns of impacts are not always consistent.^*^ 

Although, as noted, some seemingly large site impacts are not significant because of 
small sample sizes, many of the impacts that are significant should be interpreted cautiously, 
and regarded as suggestive rather than conclusive. In any evaluation, including this one, 
some impacts will be significant just by chance. In fact, even if there were no impacts, we 
would expect to find approximately 5 percent of the comparisons between treatment and 
control group members to imply significant differences at the 95 percent confidence level. 
When we adjust significance levels to account for the many comparisons that are being made 
when we derive site impacts in this evaluation, we find that most of the impacts become 
statistically insignificant. The ones that remain significant are those that are significant at the 
99 percent confidence level. 

The Cleveland site had the most consistent impacts, vrith statistically significant 
beneficial impacts on at least one outcome in each of the main categories, raising educational 
attainment and employment while lowering the prevalence of some risky behaviors. 
Cleveland was the only site to increase the percentage of enrollees who earned a high school 
diploma or G ED , raising that percentage by 19 percentage points (to 77 percent). There is 
also some evidence that the Cleveland site increased by 20 percentage points the likelihood 
of earning a high school diploma, although the significance of that impact is sensitive to the 
way that we measure high school graduation (see Appendix E). The percentage of control 
group members who earned a high school diploma or GED was particularly low in 
Cleveland (55 percent)— approximately 10 percentage points lower than the next lowest site- 
level control group, and 20 percentage points lower than most of the other site's control 
groups (see below and Appendix H). The Cleveland site also increased postsecondaiy 
enrollment and completion, increasing the hkelihood of enroUing in a 2- or 4-year college by 
18 percentage points and of earning a bachelor's degree by 6 percentage points. In addition 
to these beneficial impacts on educational outcomes, the Cleveland site increased the 
hkehhood of being employed, enrollees' hourly earnings, and the hkelihood of enrollees 
having a job with paid time off. Einally, in Cleveland, QOP had beneficial impacts on 
outcomes relating to risky behaviors and family hfe, lowering smoking and binge drinking 
rates and lowering by 19 percentage points the hkehhood of receiving welfare or food 



To adjust for random differences that may exist between the treatment and control groups within each 
site, we also obtained regression-adjusted estimates, using the method described in Appendix F. We generally 
found very similar results to those presented here; important differences are discussed below. 
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stamps at the time of the third follow-up survey (to 24 percent). The only detrimental 
impact of QOP in Cleveland was a 13-percentage-point increase in the hkelihood of 
committing a crime in the past two years (raising that to 25 percent), although the evidence 
for this impact is weak.'’® AU of these impacts, except the detrimental impact on crime, were 
significantly different from the impacts for the other six sites combined.®® 

The Philadelphia site also had several beneficial impacts, particularly relating to 
postsecondaiy educational attainment. For example, the site increased by 18 percentage 
points the likelihood of ever attending a four-year college (to 20 percent), by 13 percentage 
points the hkelihood of completing at least two years at a four-year cohege (to 13 percent), 
and by 7 percentage points the hkehhood of attending a four-year college at the time of the 
third telephone survey (to 7 percent).®^- ®® We also found that the Philadelphia site decreased 
by 23 percentage points the likehhood of receiving welfare or food stamps at the time of the 
third follow-up survey (to 21 percent). All of these impacts were significantly different from 
the impacts for the other six sites combined. In contrast to these beneficial impacts, the 
Philadelphia site had detrimental impacts on the rate of freguent binge drinking and on the 
likehhood of enrollees having been arrested or charged with a crime in the two years before 
the third fohow-up survey. 

The only other site with a beneficial impact on educational attainment was Washington, 
DC, which increased by 15 points the percentage of enrollees who engaged in any 
postsecondaiy education or training (to 69 percent) as weh as the percentage who completed 
at least two yeans of cohege or mihtaiy service, completed vocational/ technical school or an 
apprenticeship, or were honorably discharged from tiie mhitaiy, raising that percentage by 19 
points (to 39 percent).®® The Washington, DC site also had beneficial impacts on the 
prevalence of risky behaviors, lowering the freguent binge drinking rate by 12 percentage 
points (to 2 percent) and the percentage of enrollees who have at least one child with whom 
they were not currently living by 13 percentage points (to 8 percent). Ah of these impacts, 
except the impact on engagement in any postsecondaiy education or training, are 
significantly different from the impacts in the other six sites. There were no detrimental 
impacts of Q 0 P in Washington, D C. 



59 This impact is sensitive to regression-adjustment as well as to adjustments for multiple comparisons. 

“To provide some indication of how the site impacts vary and whether a site's impacts stand out from 
the impacts in the other sites, we compare each site's impact with the average impact for the other six sites 
combined. An alternative would be to compare each individual site's impact with the impact in every other 
site, but because of small sample sizes, such statistical tests would have even lower power to detect differences 
in impacts than the tests we have performed. 

61 No control group members in Philadelphia completed more than one year at a 4-year college, and none 
were currently enrolled in a 4-year college at the time of the third follow-up survey (see Appendix H). 

62 The impact on ever attending a four-year college is sensitive to regression-adjustment; when we obtain 
regression-adjusted impacts, the effect on that outcome in Philadelphia was 16 percentage points but not 
statistically significant. The impacts in Philadelphia on the other outcomes pertaining to enrollment and 
attainment at four-year colleges are not sensitive to regression-adjustment. 

63 There is also some evidence of an increase in the receipt of a high school diploma or GED in the 
Washington, D C site: the regression-adjusted impact is 14 percentage points and statistically significant. 
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In the other sites, most impact estimates were not significant, and those that were 
statistically significant were generally detrimental. The only beneficial impact that was 
significantly different from zero in a site other than Cleveland, Philadelphia, or Washington, 
D C, was a decrease in the proportion of Q 0 P enroHees with poor self-reported health status 
in the Memphis site. All of the other significant impacts in Memphis, Houston, and Y akima 
were detrimental. These included detrimental impacts in Memphis on employment and job 
characteristics, with QOP lowering the likelihood of enroUees currently being employed by 
15 percentage points and decreasing their job tenure and weeks and hours worW. The 
likehhood of being a single parent increased in Memphis by 26 percentage points (to 52 
percent), while the likelihood of receiving welfare or food stamps increased by 13 percentage 
points (to 37 percent). The detrimental impacts in the Houston and Yakima sites mostly 
pertained to risky behaviors and family life. QOP increased binge drinking rates in both the 
Houston and Yakima sites. In Houston, QOP also increased tobacco use, illegal drug use, 
and the receipt of welfare or food stamps. None of the impacts in Fort Worth was 
significantly different from zero.“' 

These impacts are broadly consistent with the pattern of impacts obtained before. 
Previously, Schirm and Rodriguez-Planas (2004) found that Cleveland and Philadelphia had 
beneficial impacts and no detrimental impacts. At that point in time, Washington, DC had 
mixed impacts, with a beneficial impact on illegal drug use but detrimental impacts on binge 
drinking and on the likelihoods of being a teen or single parent. The other sites had no 
significant impacts or mostly detrimental impacts, as we find here. 



^ Grouping sites by funding source, we find that the five DOL-funded sites collectively had a 
combination of beneficial and detrimental impacts while the two Ford-funded sites had only insignificant or 
detrimental impacts. When we examine high school completion or employment and earnings we find no 
significant impacts in either the Ford- or D 0 L-frmded sites. Considering impacts on postsecondary education 
or training, we find that the DO L-frmded sites increased by 8 percentage points the likelihood of enrollees' 
ever engaging in any postsecondary education or training and increased by 7 percentage points the proportion 
of enrollees who completed at least two years of college or military service, completed vocation^/ technical 
school or an apprenticeship, or was honourably discharged from the military. However, these sites also 
increased some risky behaviors, increasing the percentage of enrollees who had been arrested or charged in the 
past two years by 4 percentage points and the percentage who had been convicted of a crime in the past two 
years by 3 percentage points. The D 0 L-funded sites also increased the rate of welfare receipt by 5 percentage 
points. The detrimental impacts in the two Ford-frmded sites included an increase in frequent binge drinking 
of 14 percentage points, an increase in the percentage of enrollees who reported being in poor health of 10 
percentage points, and an increase in the percentage of enrollees who had a child with whom they were not 
living of 14 percentage points. As was seen in the DO L-frmded sites, the Ford-funded sites also increased 
criminal activity, raising the percentage of enrollees who had committed a crime in the past three months by 7 
percentage points and the percentage who had been arrested or charged in the past two years by 12 percentage 
points. The impacts on self-reported poor health and on the percentage of enrollees who did not live with at 
least one of their children in the Ford-funded sites were significantly different from the impacts for the D 0 L- 
funded sites. Some of the differences in impacts between the DOL- and Ford-funded sites might be 
attributable to the differences in implementation discussed below and in Maxfield et al. (2003), while funding 
source per se has, perhaps, little influence. Moreover, there were differences in implementation among the five 
D 0 L-frmded sites and differences in implementation between the two Ford-funded sites that might have led to 
the variations in impacts within each of the two groups of sites defined by funding source. 

65 Some of these detrimental impacts, particularly those in Houston, are not statistically significant when 
we adjust for random baseline differences using regression methods. 
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To assess whether impacts might be related to fidelity to the QOP model, we examined 
whether the impacts are different for sites that were more or less successful at implementing 
the model.®® As discussed above, Maxfield et al. (2003a) found that two sites— Fort Worth 
and Houston— implemented a version of QOP that deviated substantially from the QOP 
model, while the other five sites implemented versions that deviated moderately from the 
model. 

Estimating impacts separately for these two groups of sites, we find that there are 
almost no differences in impacts (see Tables 25-28). Although the estimated impacts might 
tend to be more beneficial or less detrimental in the sites that deviated only moderately from 
the QOP model compared with the sites that deviated substantially from the model, neither 
group of sites had many beneficial impacts, and the impacts for the two groups of sites were 
generally not significantly different from each other. One exception is for employment in 
the past 12 months, on which the sites that deviated substantially from the QOP model had 
a significant 8-percentage-point beneficial impact whereas the other sites had a statistically 
insignificant 4percentage-point detrimental impact. This may be partially due to an increase 
in postsecondaiy education and training in heu of employment among enrollees in the latter 
group of sites, but the impacts on postsecondaiy education or training are not significant for 
either group of sites. The other outcomes on which the impacts are different between the 
two groups of sites are daily tobacco use and food stamp or welfare receipt, on which the 
sites that deviated moderately from the Q 0 P model had detrimental impacts while the other 
sites had insignificant impacts. On balance, however, it does not appear that the observed 
variations in fidehty to the QO P model substantially influenced impacts. In the Discussion 
section, we review other potential explanations for differences in impacts. 



Fidelity of implementation is not a baseline characteristic. It is, instead, an outcome, an outcome that 
might have influenced and been Influenced by students' outcomes in a site. Thus, we should be cautious when 
interpreting impacts and making causal inferences pertaining to fidelity of implementation. 
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Table 21. Impacts on High School Completion by Site (Percentage Points) 
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Estimate significantly different from zero at the 90% confidence level, two-tailed test 
Estimate significantly different from zero at the 95% confidence level, two-tailed test 
Estimate significantly different from zero at the 99% confidence level, two-tailed test 




■o 

o 

(Q 

O 

'•B 

c 

o 

(0 



(0 

(0 

0) 

c 

3 

</T 

c 

o 

Q. 

O 

G) 

(0 

c 

o 

o 

o 

Q. 



CO 

> 

n 

(0 

G) 

C 

c 

G 

LU 

■G 

C 

G 

C 

o 

E 

> 

o 

Q. 

E 

LU 

C 

o 

G 

O 

G 

Q. 

E 



CO 

CNJ 

o 



o 

G 

Q. 

E 



G 

E 

G 

> 



G 

in 

Q. 

<D 

■o 

_G 

!E 

CL 



in 

Q. 

E 

G 



o 

Q 



■O 

c 

_G 

G 

> 

_G 

O 



o 



G 

E 

o 

o 

O 



i- i- CM i- 



m CM CM 



00 1- 



1- 



m i- 



00 CD CM 



~0 T_ 

G o 

l-gS 

E c 3 

G 3 O 

c c c 
a> G G 

3 3 3 

O O O 



in 



c 

G 

O 

O 

o 

G 

O 

G 

o 

> 

g" 

D) 

= ^ 
O G 
O 

S 1 
o a> 
"D ^ 

gl 

c C 
a> G 

si 



o 

CM O CM 



O 

CT> O O 



O CT> 



O 

00 CM O CM 



* 

CD O LO 



O 



c 

o 



G 

G 

Q. 



G 

O 



G 

JO 



3 

C 



■o 

G 



■D 

G 

>, 

O 

Q. 



G 

> 

LU 



^ 8 - 
o E 

E ^ 



G 

tf5 I 

11 

CD G 
"c 

o S 

Q. O 
E G 

LU CL 



CM IT) 



CD 00 



C 

O w 

CM o 

H £ 

'Tn ^ 

^ G JO 
G Q-.O 

I ■- c 

^ G G 
O ^ t 

S 2 « 

E <1> G 

i 

s = s 



c 

G 



3 

O 



G 

G 



G 

■o 

G 



3 

O 



M- ^ 

o CO 
0> 

-p G _G 
C O 

G ^ 

I 

3 .0§ 



o 

CM 



CM 

CM 

IT> 



1^ CO 
CM CO 



CD 

CT> 

CD ' 

cm" 



o 

CJ> o 
' 

ir> 



o 



CM 

CD 

CO 



O . 

CM 

CM 



CM 
00 
CJ> o 
ir> ' 
CM^ 

cm" 



JO 

.o 

c 

G 

3 

O 

c 

'g 



G 

G 

G 



G 

Q. 



3 

O 



G 

^ = 
G O 

= -o 

O ^ 
O 



G • 

SZ 

c 

o 

E 

CM 



G 

G 

Q. 

C 

G 

D) 

C 






m CO CT> 



G 

c 

G 

JO 

c 

G 

E 

G 



C 

G 

3 

O 

c 

'g 

E 

CO 

G 

O) 



G 
O 
C 
G ; 

3 

G 

C 



g:p g 
G G ^ 
JI G 



C 

_ 'c 

E 

G d> 

S ^ 

O o 



^ JO JO 

o o o 



G ^ 

3 G 

"o' 

S =3 

CO <5 

£ g 2 

ra o ■£ 

CD -£= o 

.Egg 
G G 

o CO ^ 
CD CD 
o3 2 Q_ 

5 = 0 

CO “ O 

B 

E -2 S 

III 

G 

*“ B 

c CD Q- 

G G E 

8 8 « 
E o CO 

G O 



3 

p 

D) c/i 

□_ -B 

OB 

o ^ 

Q3 ^ 



G 

>^ 

G 

> 

3 

G 



C 1= 

Q> «s 
G G <1> 

E R 
^ n O 



y- C Q. 

o ^ 

^ G G 
O) ^ ^ 

III 

c k o 
o n a> 

o ^ G 
CD E 

O) C U- 
r- G 
■Q 

^ C - 
G G C 
O 
G G 

col i. 

■g.E 

TJ Q-^ 
G G S 

■g E c 

G 
■Q 



O 



3 



JZi 



G 


G 


G 


<l> 


G 


CD 








Ti 


Ti 


■n 


G 


G 


G 


'g 


'g 


'g 




-*Y' 




o 


o 


o 














"g 


Td 


"CD 


> 


> 


> 


_G 


_G 


_G 


<1> 


CD 


CD 


O 


C> 


O 


r 


r 


r 


CD 


CD 


CD 


O 


O 


O 


*r 


*r 


*r 


o 


o 


o 


o 


o 


o 




<v^ 




C 5 


U> 




0^ 


0^ 


0^ 


CD 


CD 


CD 


JO 


sz 


r 














G 


G 


G 


G 


G 


G 


CD 


CD 


CD 








G 


G 


G 








CD 


CD 


CD 


jr 


JZ 


JZ 








o 


o 


o 


"g 


"g 


"g 








o 


o 


o 














c> 


c> 


c> 


m 


m 


m 


o 


o 


o 


E 


E 


E 


G 


G 


G 



E E E 
o o o 



E "P c 



>% 


G ^ 


i_ 


a; a; a; 


G 

> 


s s 

> G 


3 

G 

G 


cE cE cE 


3 




G 




G 


O 


E 




G 


G G 






c 


Q. G 


G 


c c c 


o 


E ^ 


G 


G G G 


SI 

Q. 


^ C 
G 
r~ 4— 


E 

o 


O .O .O 


G 


O 


o 


'c 'c 'c 


G 


G ^ 


3 


0)0)0) 


1- 

G 

p 


LU ^ 


o 


(7) O) ^ 


3 


G 




H — 


O 


O 




H — H — 


(/) 


z 







* 

* 

* 



58 



Estimate significantly different from zero at the 90% confidence level, two-tailed test 
Estimate significantly different from zero at the 95% confidence level, two-tailed test 
Estimate significantly different from zero at the 99% confidence level, two-tailed test 
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Estimate significantly different from zero at the 90% confidence level, two-tailed test 

Estimate significantly different from zero at the 95% confidence level, two-tailed test 

Estimate significantly different from zero at the 99% confidence level, two-tailed test 




Table 25. Impacts on High School Completion by Site’s Deviation from OOP Model (Percentage Points) 



W 

O 

CO 

Q. 

E 



_o> 

CL 



CO 

c/} 



Si 

2 

0) 

p 

^ o 

.55 O 

So 

Q E 
to 2 

SZ 

(/) 



CD 

■o 

O 

CD □_ 

to O 

■>o 

CD ^ 

Q E 

^ o 

CO 



c/) CO 

CO 

3 

C/) 



O 

o 

"3 

O 



CO 

E 

o 

Q. 

cn 



■O 

CD 

> 

’CD 

O 

CD 

CC 



Q 

LU 

o 

O 

CO 

E 

o 

Q. 

C/) 



■D 

CD 

> 

'CD 

O 

CD 

CC 



O? 

CO 



(/) 



(/) 

ff 

(/)(/) 

CD 2 

■5>| § 

Q. 

CD CJ> 

JB ” 

o ^ 

2 ^ 

CD (D 

CD 



'CD 

O) 






(/) 



c 

'cO 



CO 



CD 4 -» 

CD £ O 

to 00 
g o m 
■.S "CJ 

LU 

CO 

^ Q- 

^ CD C 

CO c/) c 

CD - 

E _ ^ 

(/) o 

CD 

O CO 
O) 05 

cl-B 2 

Q W 05 

O'o 5 ^ 



c to 
o 05 
Q. c 

Q_ (/> - 



O 

CD 

C:i 



O <1> 



c 

o 



o 



(/) • 



o 

(/) 

c 

CO 



■O 

c 

CO 

(/) 

>^ 

CD 

£ 

3 

(/) 

CD 

c 

o 

SI 

Q. 

_0> 

CD 



CD 

O 

3 

o 

C/) 



(/) s. 



c -D 
CD CD 
CD • 



3 

CD CO 
- CD 



CO C/> ^ 
Q. CD (/) 

E g I 
^ E o 
o 0> 

CO 

LU ‘-0 



s 

o 



3 

o 



(/)(/)(/) 

B B 3 

■O "D "O 

_o> _o> _ 0 > 

CO CO CO 

6 6 6 



CD CD CD 
> > > 
Si Si Si 

CD CD CD 
O O O 
C C C 
CD CD CD 
■D "D "O 



C C 
O O 



CO CD CD 
CD (/) 

E ^ 

Q. _ Q. 
3 .E CD 

2 05 05 

O CD C 
2-0 0 
c o 

E O CD 
Q Q_ Ui 



CD ^ C 
-C c CO 

o - 
O) c ^ 

-O 

o c ^ 

CO CO C 

^ o 

^ (/) w 

c 0 ) 

C/> CD Q. 
-o Q-^ 

CD C/) S 

■g 2 c 

CD 

■o 



(/)(/)(/) 
CD CD CD 



O m CT> 
CJ> CT> CT> 

CD CD CD 
^ ^ SI 

CO CO CO 

2 2 2 
0) 0) 0) 
Si Si Si 

E E E 

CD CD CD 

2 E 2 
Si Si Si 

CL Q. CL 

E E E 

CO CO CO 
(/)(/)(/) 

0) 0) 0) 
S S "B 
o o o 

CO CO CO 
c c c 
o o o 

o o o 
CO cO CO 
Q. Q. Q. 

.i .i -i 

CD CD CD 

B B 

E E E 
o o o 



c c c 

CD CD CD 



■O "O "O 

C C C 
CO CO CO 

o .o ,o 

'c 'c 'c 
^ ^ O) 

iy) O) iy) 



■O "D "O 

Si Si Si 



O O O 
"a> 

> > > 

Si Si Si 

CD CD CD 
O O O 
C C C 
CD CD CD 
■O "O "O 



C C 
O O 



o m CT> 
CJ> CJ> CT> 
CD CD CD 



CO CO CO 

o o o 



N N 

E E 
o o 



c c c 

CD CD CD 



■O "D "O 

C C C 
CO CO CO 

o .o ,o 

'c 'c 'c 
D) ^ D) 
’(/) ’(/) '(/) 
CD CD CD 
CO CO CO 

E E E 

CO CQ CO 
LU LU LU 



60 




Table 26. Impacts on Postsecondary Attainment by Site’s Deviation from QOP Model (Percentage Points) 
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Table 27. Impacts on Employment and Earnings by Site’s Deviation from QOP Model (Percentage Points, Unless Otherwise Indicated) 
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Estimate significantly different from zero at the 90% confidence level, two-tailed test 
Estimate significantly different from zero at the 95% confidence level, two-tailed test 
Estimate significantly different from zero at the 99% confidence level, two-tailed test 
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Estimate significantly different from zero at the 90% confidence level, two-tailed test 
Estimate significantly different from zero at the 95% confidence level, two-tailed test 
Estimate significantly different from zero at the 99% confidence level, two-tailed test 




Summary of Impacts 



Several findings emerge from our analysis of the data collected in the final survey of the 
evaluation: 

• QOP did not achieve its first primaiy objective. It did not increase the likelihood of 
graduation from high school with a diploma. It also did not increase the likelihood of 
high school completion by earning either a diploma or G E D . 

• Q 0 P has not achieved its second primaiy objective. It has not increased the likelihood 
of ever engaging in postsecondaiy education or training, including college, 
vocational/ technical school, an apprenticeship, or the mUitaiy. Furthermore, QOP has 
not increased persistence in such activities and thus, attainment of postsecondaiy 
education or training. Although data collected earlier in the evaluation indicated that 
QOP was increasing rates of entiy into postsecondaiy education or training when 
sample members were in their late teens and early twenties (Maxfield et al. 2003b; 
Schirm and Rodriguez- Planas 2004), the most recently collected data reveal that this 
impact was not sustained in the longer run as sample members entered their mid- 
twenties. 

• Q 0 P has not improved employment- related outcomes. Improving such outcomes was 
a principal motivation for QO P's two primaiy objectives and, thus, an implicit long-run 
goal of the program. However, when sample members were entering their mid- 
twenties, QOP had not increased their likelihood of being employed, the fraction of 
time employed, annual or hourly earnings, or the likelihood of having a job with 
benefits such as health insurance, paid time off, or pension and retirement benefits. 

• Q 0 P has not generally achieved its secondaiy objective of reducing the broad range of 
risky behaviors targeted by the program. When sample members were in their late 
teens, Q 0 P did not reduce any risky behaviors, such as binge drinking, illegal drug use, 
crime, or teen parenting (Maxfield et al. 2003b). Although it still did not reduce binge 
drinking or crime when enrollees were in their early twenties, QOP did reduce illegal 
drug use (Schirm and Rodriguez-Planas 2004). More recently, according to data 
collected when sample members were entering their mid-twenties, QOP has not had 
any such beneficial effects in reducing substance abuse but has had detrimental effects 
on crime and involvement with the criminal justice system, increasing by 3 percentage 
points the likelihood of committing a crime in the three months before the most recent 
survey and by 6 percentage points the likelihood of being arrested for or charged with a 
crime in the two years before the survey. 

However, some sites and subgroups of students showed some promising results. 

• QOP seems to have been more effective for younger enrollees than for older enrollees. 
QOP increased rates of high school completion and engagement in postsecondaiy 
education or training among younger enrollees (the two -thirds of enrollees who were 
age 14 or younger when they entered the ninth grade), but it had no such impacts on 
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older enroHees (those age 14 or older when they entered the ninth grade). Although 
most differences between subgroup impacts are not statistically significant because of 
small sample sizes, we observed a significant difference between impacts for younger 
and older enroHees in the likelihood of receiving a high school diploma. 0 ther findings 
pertaining to subgroup impacts include the foHowing: 

- QOP had some impacts on both females and males but does not seem to have 
consistently benefited one group more than the other. For females, QOP reduced 
the Hkelihood of engagement in postsecondaiy education or training at the time of 
the last survey but increased the Hkelihood of having a job with benefits such as 
health insurance or pension or retirement benefits. For males, QOP had one 
beneficial impact— it increased the likelihood of receiving a high school diploma or 
GED, although the evidence is not strong. QOP also had detrimental impacts on 
males. It decreased the likelihood of having a job with benefits and increased the 
likeHhood of criminal activity and involvement with the criminal justice system. 

- QOP increased rates of high school completion and engagement in postsecondaiy 
education or training among younger enroHees but not among older enroHees, as 
noted above. For younger enroHees, QOP increased by 7 percentage points the 
likeHhood of receiving a diploma and by 6 percentage points the Hkelihood of 
receiving a diploma or GED. It also increased by 10 percentage points both the 
Hkelihood of ever engaging in postsecondaiy education or training and the HkeHhood 
of completing at least two years of coHege or the miHtaiy, completing 
vocational/ technical school or an apprenticeship, or being honorably discharged 
from the mHitaiy. Only the impact on diploma receipt among younger enroHees is 
significantly different from the impact among older enroHees. 

- Q 0 P had few significant impacts and no strong, consistent pattern of impacts across 
subgroups defined by rank in the eHgible baseline grade distribution. (The eligible 
grade distribution was based on grade point average in the eighth grade and excluded 
youth who were ineHgible for Q 0 P because their grades were too high.) 

• QOP's impacts appear to vaiy widely by site, although many seemingly large impacts 
and seemingly large differences in impacts are not significant because of smaH sample 
sizes. Findings pertaining to site impacts include the foHowing: 

- The Cleveland, PhHadelphia, and Washington, DC sites had mostly beneficial 
impacts. 

- With the exception of a decrease in the proportion of enroHees with poor self- 
reported health status in the Memphis site, none of the other four sites— Fort 
Worth, Houston, Memphis, or Yakima— had beneficial impacts whHe some had 
detrimental impacts. 

- The Cleveland site was the only site that increased high school completion, raising 
the HkeHhood of earning a diploma or GED by 19 percentage points. The Cleveland 
site also increased the HkeHhood of ever attending a two- or four-year coHege by 18 
percentage points and the HkeHhood of earning a bachelors degree by 6 percentage 
points. In addition to beneficial impacts on some employment- related outcomes and 
on smoking and binge drinking rates, the Cleveland site reduced by 19 percentage 
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points the likelihood of receiving welfare or food stamps. A 13-percentage-point 
increase in the hkehhood of committing a crime in the two years before the most 
recent survey was the Cleveland site's only detrimental impact. 

The Philadelphia site had beneficial impacts on postsecondaiy educational 
attainment increasing by 18 percentage points the likehhood of ever attending a 
four-year college and by 13 percentage points the hkehhood of completing at least 
two years at a four-year college. Although the Philadelphia site had detrimental 
impacts on the rate of freguent binge drinking and the likehhood of being arrested or 
charged with a crime in the two yeans before the most recent survey, it reduced by 23 
percentage points the hkehhood of receiving welfare or food stamps. 

The Washington, DC site increased postsecondaiy education or training, raising by 
15 percentage points the hkehhood of ever engaging in postsecondaiy education or 
training and by 19 percentage points the likehhood of completing at least two years 
of cohege or mhitaiy service, completing vocational/ technical school or an 
apprenticeship, or being honorably discharged from the mihtaiy. The Washington, 
D C site also had beneficial impacts on the rate of freguent binge drinking and the 
likelihood of having a child with whom the enrollee was not hving. 
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Discussion 



Four questions motivate this discussion: 

1. Why didn't Q 0 P have larger impacts overall? 

2. Why were the results in the demonstration different from those of the pilot study? 

3. Why were there impacts for some groups but not for others? 

4. What lessons does this provide for the future? 



Why Didn'tQOP Have Larger Impacts Overall? 

We might have expected QOP to have large impacts. One reason for high expectations is that 
some results from the pilot study were seen as promising, leading D 0 L and the Ford Foundation to 
initiate the larger-scale demonstration. Another reason is that although there were some potential 
problems, QOP avoided two design flaws that have been common to other programs for 
disadvantaged youth: (1) as a result of their recruiting and apphcation procedures, most programs 
serve motivated students who are likely to do well even in the absence of a program, and (2) their 
program interventions are weak. Both flaws are often cited in explanations for why many youth 
programs have failed to achieve large impacts. However, with respect to the first flaw, QOP was 
designed to serve all ehgible students or, with the evaluation, a random sample of eligible students. 
QOP did not have an application process that would "weed out" less motivated students or any 
other screening procedures— such as requiring recommendations— that would "cream" students 
likely to perform well without any assistance. With regard to the second flaw, QOP was more 
intensive and comprehensive than most programs, even when not implemented with full fidehty to 
the model. 

Despite its promise, QOP achieved no success overall. Next, we speculate about why it was 
not more successful. We can only speculate because the evaluation did not control for the factors 
discussed below, such as variations in implementation and participation. Therefore, we cannot state 
conclusively or even with much confidence that QOP would have had large impacts had it done 
something differently. Thus, this discussion of our findings must be taken as suggestive rather than 
conclusive, motivated to prompt discussion and further research. 

Implementation Problems. The demonstration sites had difficulty implementing the full 
QOP model, with five sites deviating from the model moderately and two sites deviating from it 
substantially, as discussed above. Eveiy site had only limited success in implementing the education 
component of the model, which is consistent with the lack of impacts on education-related 
outcomes such as high school completion and postsecondaiy enrollment. Tutoring was poorly 
implemented by aU of the sites, and most sites did not develop formal, comprehensive individuahzed 
education plans for enrollees. Even though case managers monitored and sought to improve class 
attendance and course grades— activities that might have kept enrollees from failing some courses 
and enhanced enroUees' prospects for graduating— we found previously that QOP did not improve 
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enrollees' high school grades, increase the number of credits earned, or raise achievement test scores 
(Maxfield et al. 2003b). 

With no beneficial effects on these indicators of academic achievement, QOP might still have 
increased the hkehhood of graduation through intensive mentoring and case management. A main 
objective of the mentoring/ case management component, which was much better implemented 
than the education component, was to keep enroUees focused on overcoming barriers to the goal of 
graduation (and the goal of enrollment in postsecondaiy education or training). Case managers 
attempted to prevent each enrollee from giving up on school, advocated on behalf of the enrollee 
with the school, and tried to protect the enroUee from outside distractions and responsibilities that 
would divert the enrollee's attention from school. However, these efforts were apparently not 
sufficient for the average enroUee. 

The lack of beneficial impacts on risky behaviors might not be surprising given that QOP did 
not begin until enroUees were already in high school and engaging in risky behaviors. Even though 
QOP's developmental component was implemented much more successfuUy than some other 
components, it was geared to preventing risky behaviors rather than remediating the effects of the 
risky behaviors in which many enroUees had already engaged before entering QOP (Maxfield et al. 
2003a). At the beginning of the demonstration, sites underestimated enroUees' needs for supportive 
services such as child care, substance abuse treatment, and famUy counseUng, and, later, they 
struggled to address those needs. Thus, QOP might not have been able to break, for example, an 
ongoing pattern of substance abuse. It is also possible that any positive effects of the preventive 
measures undertaken by QOP were offset by the negative peer effects introduced by bringing 
enroUees together for program activities. 

Despite sites' problems in implementing the QOP model, we did not find differences in 
impacts between sites that deviated moderately from the model and those that deviated substantiaUy 
from the model. Though a fairly crude comparison, our finding suggests that implementation 
problems alone do not explain QOP's lack of success. 

Low Levels of Participation. Another explanation for QOP's limited impacts could be the 
low levels of participation relative to the target set by program developers. The target was set as a 
standard of participation that would, presumably, enable enroUees to attain Q 0 P's objectives. 

Although QOP enroUees spent substantiaUy more time in program activities than did 
participants in the typical JTPA youth program, for example, the number of hours spent in program 
activities during the first year by the average QOP enroUee fell substantiaUy below the program goal. 
Over time, hours of participation feU for the average enroUee while the proportion of enroUees with 
Uttle connection to the program grew steadUy, as discussed above. Relative to enroUees' high school 
experiences, participation in QOP and, specificaUy, the program's educational activities, might have 
been insufficient to substantiaUy affect educational outcomes. Furthermore, low participation 
during the last year or two of the demonstration may have meant that few enroUees were 
substantiaUy engaged in QOP at precisely the time when those who had not dropped out of high 
school were nearing the end of high school and considering postsecondaiy activities. In the fourth 
year, when enroUees were scheduled to be in twelfth grade, average participation in QOP was 103 
hours per enroUee— 14 percent of the target— and about one-guarter of enroUees were not 
participating at all. The enroUees who were stUl participating actively at the end of the 
demonstration may have been the most motivated and likely to succeed even in the absence of 
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QOP. Under such conditions, it might have been difficult for QOP to influence the choices made 
by many enroHees regarding postsecondaiy activities. 



These findings lead to the guestion of whether higher levels of participation cause larger 
impacts. However, it is difficult to answer that guestion conclusively. The only experimentally 
controlled factor in the demonstration was whether a student was enrolled in QOP. We could not 
control other factors, such as the level of enroHees' participation. Therefore, because we cannot 
construct appropriate comparison groups, we cannot estimate the impact for enroHees with high 
participation or the impact for enroHees with low participation.®^ 

What we can do, however, is examine the average outcomes of QOP enroHees with higher and 
lower levels of participation; if we do not observe large differences, we might conclude that we 
would be unhkely to find differences in the impacts for these groups. However, we find that QOP 
enroHees with higher levels of participation do tend to have higher educational attainment and 
higher levels of employment and earnings than enroHees with lower levels of participation (see Table 
29). Thus, it is possible that QOP had large impacts on the group of enroHees with higher 
participation. However, we also see that enroHees with higher levels of participation in QOP also 
had different baseline characteristics from those with lower participation levels. In particular, the 
enroHees with higher participation had higher grades than those with lower participation. We 
observed only a smaH set of basehne characteristics for our evaluation sample, and it is likely that 
these two groups also differ on unobserved characteristics such as motivation and parental support. 
We thus cannot identify whether the differences in outcomes are attributable to QOP or to 
differences in the pre-existing characteristics of the sample members with higher and lower 
participation levels. 

Limited Influence on EnroUees' Schools and Classroom Experiences. QOP was not 

designed to influence the structure, poHcies, or operation of the high schools with which the local 
QOP programs were associated. The lack of influence on those schools might help explain the lack 
of impacts on education-related outcomes such as high school completion. QOP could not 
substantiaHy influence what went on in the schools, which was where enroHees were supposed to 
spend most of their time and receive most of their academic instruction. The supplement^ activities 
that QOP provided might not have been sufficient to overcome the problems in the schools and 
enable enroHees to graduate from high school and succeed in postsecondaiy education. 

Why Were the Results in the Demonstration Different Erom Those of the Pilot Study? 

Carried out between 1989 and 1994, the pHot study operated in five sites (Milwaukee, 
Oklahoma City, Philadelphia, San Antonio, and Saginaw). In each site, 50 eligible rising ninth-grade 
students were selected to participate in the pilot study, with 25 initially assigned at random to the 
treatment group and 25 to the control group. The results of the study were based on surveys 



To estimate impacts for enroHees with high levels of participation, for example, we would need to apply quasi- 
experimental methods to identify a comparison group of control group members who would have had high levels of 
participation had they been selected to enroll in Q 0 P (Frangakis and Rubin 2002). However, we lack sufficient baseline 
data for that purpose. 



69 




Table 29. Comparison of Higher Participation Enrollees with Other Enrollees (percentages) 





Higher 

Participation 


Lower 

Participation 


Difference 


Outcomes 


Received High School Diploma 


88 


51 


37*** 


Received High School Diploma or GED 


94 


67 




Completed at Least 2 Years at a 4-Year 


College 


32 


5 


2"^*** 


Completed at Least 2 Years at a 2- or 4-Year College 


45 


15 


30*** 


Currently Employed 


71 


66 


4 


Works at Least 25 Hours per Week at Main Current Job 


43 


59 


-16* 


Binge Drinking in Past Month 


28 


39 


-10 


Committed a Crime in Past 2 Years 


8 


21 


-13* 


Currently Receiving Welfare or Food Stamps 


24 


31 


-6 


Baseline Characteristics 


Male 


62 


53 


10 


Grades in Bottom Third 


19 


43 


24*** 


Grades in Top Third 


52 


27 


24** 


Age < 1 4 


75 


63 


12 



Source: MIS data and telephone survey. 

Note: Higher participation is defined as participating in more than 1 ,500 hours of OOP activities during the 

demonstration. Approximately 13 percent of all OOP enrollees participated at that level. All other 
enrollees were classified as having lower participation. 

Because the higher and lower participants are nonrandom samples of the full set of enrollees, differences 
in means between the two groups cannot be interpreted as the causal effect of higher versus lower 
participation. 

* Estimate significantly different from zero at the 90% confidence level, two-tailed test 

** Estimate significantly different from zero at the 95% confidence level, two-tailed test 

*** Estimate significantly different from zero at the 99% confidence level, two-tailed test 



conducted with students within the first four years after entry into the program, with the final survey 
conducted in fall 1993, a few months after the students' scheduled high school graduation. At the 
end of the four-year pilot, the study reported beneficial effects of Q 0 P on a variety of outcomes, 
including reductions in the high school dropout rate and teen parenting and increases in high school 
completion and enrollment in two- and four-year postsecondary institutions (Hahn, Leavitt, and 
Aaron 1994). Results varied substantially across sites, however, with most of the pilot-wide impacts 
attributable to a single site (Philadelphia), and few significant impacts found in any of the other sites. 
In fact, the Milwaukee site was completely dropped from aU analyses because of poor 
implementation.®’^ 

In addition to noting the problems of implementation and the variations in impacts across sites, 
technical reviewers raised concerns about the methods used by the pilot evaluation and the 
implications for the findings. Thus, while encouraged by some of the results from the phot, D 0 L 



68 Inclusion of aU of the sites originally participating in the pilot study (including Milwaukee) would likely have led 
to smaller estimated effects in the pilot study. 
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and the Ford Foundation began planning in spring 1995 a second test of the program— the QOP 
demonstration. As documented in this report, we have not found for the demonstration the large 
positive effects that were found in the pilot. 

T 0 understand why the results from the pilot and demonstration are seemingly so different 
reguires an understanding of some key differences in the design and implementation of the two 
studies. 0 ne potentially important difference is that the high schools in the D 0 L-funded sites in 
the demonstration were reguired to have high dropout rates (over 40 percent). No such 
reguirement pertained to the Ford-funded sites or tiie pilot. Another difference is that the 
demonstration added a fifth year to the program, both to assist enroUees who had not graduated 
from high school on time and to help with the transition to postsecondaiy education for those going 
on to further schooling. 

A potentially critical difference was the type of students served by the program. The pilot 
targeted low-income students while the demonstration targeted low-performing students. In the 
pilot, students were eligible for QOP if their families received public assistance. In contrast, 
students were eligible for the demonstration if their eighth-grade GPA placed them in the bottom 
two-thirds of the grade distribution for entering ninth graders. 

These differences in the program eligibility criteria are reflected in the baseline characteristics of 
students in the two studies. In the pilot, 42 percent of students had a B average or better in eighth 
grade, and 83 percent had a C average or better (Hahn, Leavitt, and Aaron 1994; Table 1989-A). In 
contrast, the median eighth-grade GPA among students in the demonstration was 1.8 (a C- 
average).®® A similar difference was evident in the Philadelphia site. In the pilot, 52 percent of 
students in Philadelphia had a B average or better in eighth grade. In contrast, students in the 
demonstration in the Philadelphia site had a median eighth-grade GPA of 71, corresponding to a 
C-.™ Based on their experiences working with the two sets of QOP enrollees, the staff in the 
Philadelphia site who were involved in both the pilot and the demonstration reported that the 
academic needs of Q 0 P enrollees were much greater in the demonstration than in the pilot. 

There were also some differences between the levels of participation by QOP enrollees in the 
pilot and demonstration, with average levels of participation higher in the pilot than in the 
demonstration. In the pilot, enrollees averaged 315 hours per year in the first two years. The 
average in Philadelphia, the pilot site with the highest level of implementation fidelity, was 542 hours 
while the other three sites in the analysis averaged 239 hours. In comparison, across all 
demonstration sites, the level of participation averaged 214 hours per year in the first two years. As 
in the pilot, the Philadelphia demonstration site had the highest level of participation, with an 
average of 323 hours per year in the first two years. The other sites in the demonstration averaged 
195 hours per year during that same period. 

Another potentially important difference between the two studies is their timing, with the pilot 
starting six years before the demonstration. During the period between the studies, there might 
have been substantial changes in schools, the economy, and other factors influencing students' 



® This average pertains to the five sites with G PAs caicuiated on a four-point scaie. In the two sites with a 100- 
point scale— Fort Worth and Philadelphia— the median G PAs were 82 and 71, respectively. 

’0 Information for converting from one scale to another was obtained from the NAEP High School Transcript 
Study (U.S. Department of Education, National Center for Education Statistics 2004). 
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success during and after high school. However, it is unknown what effect such changes might have 
had on the impacts of QOP. Although comparing the outcomes of control group members in the 
pilot and demonstration studies could, in principle, shed some hght on this issue, differences in the 
timing of the follow-up surveys in the two studies, as well as differences in the precise definitions of 
high school completion and postsecondaiy engagement, make that comparison infeasible.’^ 

G iven the differences between the two studies, it is difficult to reconcile the differences in their 
impacts. The demonstration, however, builds on the pilot and addresses some of its hmitations, 
such as the small sample size. The demonstration provides the best estimates of the fairly recent 
effects of Q 0 P on a relatively large sample of low-performing students. 

Why Were There Impacts for Some Groups hut N ot for Others? 

D espite the lack of overall impacts, we do find promising results for a few groups of enrollees, 
particularly younger enroUees— those who likely had not been held back a year before high school— 
and enrollees in Cleveland, Philadelphia, and Washington, D C. In interpreting the subgroup and site 
impacts, however, we must be cautious because, in deriving so many estimates, we will obtain some 
significant impacts just by chance. When we adjust significance levels to account for chance 
impacts, we find that most of the impacts become statistically insignificant. Thus, these impacts 
should be regarded as suggestive rather than conclusive. Nonetheless, examining them might help 
reveal when and for whom Q 0 P is potentially effective. 

Some of the variation in site-specific impacts might result from site-by-site variation in the 
depth of understanding of the QOP approach to youth development, the background and training 
of the case managers and coordinator, the style of mentoring QOP enroUees, or other such factors. 
For example, that the PhUadelphia site had some relatively large impacts is consistent with several of 
the site's characteristics. One such characteristic is that, from the outset, the QOP staff in 
Philadelphia understood the complex and nontraditional QOP model, especially the education 
component, and were able to implement it more effectively and guickly than staff in other sites. 
Many QOP staff in other sites regarded QOP as substantiaUy different from other programs 
operated by their CBO s. They reported that they needed at least one year, two training conferences, 
and ongoing technical assistance to understand the model and how to implement it. Staff in the 
Philadelphia site, including one of the original designers of the QOP model, provided technical 
assistance to the other sites. 

The prominent role of the Philadelphia CBO in designing and, later, marketing the Q 0 P model 
might have given the site a substantial stake in the success of the demonstration and might have led 
its management to invest greater resources than did the management of other sites. For example, 
the Philadelphia site spent more than twice as much per enrollee as any other site, and a case 
manager in the PhUadelphia site received substantiaUy higher compensation than a case manager in 
any other site. The higher level of compensation in the Philadelphia site might have produced more 
effective case management by, for example, encouraging case managers to devote extra time to 
QOP activities.” However, while the impacts in Philadelphia were different from the impacts in the 



According to Maxfield et al. (2003b), a substantial fraction— nearly 10 percent of control group members— had 
earned a GED within approximately six months of their scheduled high school graduation. However, it is not clear 
whether Hahn, Leavitt, and Aaron (1994) included GED completion in their measure of high school completion. 

Although variations in staff compensation, staff background and training, and mentoring style, for example, 
might have caused some of the variations in site impacts, the Q 0 P demonstration was not designed to measure the 

(continued) 
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other six sites combined for outcomes pertaining to four-year college attendance and welfare or 
food stamp receipt we do not find evidence of strong, broad relationships across all outcomes and 
aU sites between the guahty of implementation and the level of impacts, as discussed in detail above. 

Another possible explanation for the differences in impacts across sites could be different levels 
of participation by enroHees. However, we find httle evidence supporting such an explanation. The 
sites with the lowest levels of participation— Houston and Fort Worth— are also the sites that 
deviated substantially from the Q 0 P model, and we did not find significant differences in impacts 
between those two sites and the remaining five sites. The two sites with the highest levels of 
participation— Philadelphia and Y akima— were the two sites funded by the Ford Foundation. As 
reported above, we found that the D 0 L-funded sites generally had more beneficial impacts than did 
the Ford-funded sites. Both analyses suggest a weak relationship between a site's average level of 
participation and size of impacts. 

It is also important to remember that impacts are affected by not only enroUees' experiences 
with QOP but dso the broader conditions in which the programs operate. In Cleveland, for 
example, where we found beneficial impacts on a variety of outcomes, the students in the control 
group had particularly poor academic and employment outcomes relative to control group members 
in the other sites. In Cleveland, just 35 percent of control group members earned a high school 
diploma, while the average in eveiy other site was at least 44 percent and generally above 60 percent. 
Likewise, the fraction of the control group in Cleveland that was employed at the time of the survey 
was the lowest in the demonstration. Fewer than half of control group members in Cleveland were 
employed at the time of the most recent survey as compared with at least 58 percent in every other 
site. Showing a similar pattern, control group members in Cleveland were more hkely to engage in 
risky behaviors such as smoking and binge drinking as compared with control group members in 
other sites. Thus, the conditions in Cleveland, where youth eligible for QOP were likely to 
experience especially poor outcomes without intervention, might have created an opportunity for 
Q 0 P to have relatively large beneficial effects. 

What Lessons Does This Provide for the Future? 

There are dramatic discrepancies in high school completion and coUege-going rates between 
more and less advantaged students. Among all eighth graders in 1988, 95 percent of those in the 
highest guarthe of socioeconomic status graduated from high school within 12 years whUe only 64 
percent of those in the lowest guartile did so. College completion rates are even more disparate, 
with 51 percent of students in the highest guartile of socioeconomic status earning a bachelor's 
degree within 12 years after eighth grade as compared with just 7 percent of those in the lowest 
guartHe (U.S. D epartment of Education 2002b). 

If policymakers seek to narrow these discrepancies, there is clearly a need for further research 
into and development of programs to improve the educational outcomes of at-risk students. The 
results obtained from the Q 0 P demonstration indicate that Q 0 P as a whole did not succeed for the 



(cDntinued) 

effects of such factors. In addition, quasi- experimental methods to separate out these effects and, more generally, 
explain variations in impacts across sites, cannot help much because the demonstration included only a small number of 
sites, and they differed in so many ways that we cannot disentangle the effects of their differences. For a discussion of 
how the impacts of mentoring programs might be associated with various indicators of mentoring style, see Rhodes et al. 
( 2002 ). 
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broad range of youth targeted by the demonstration. Because the demonstration was not designed 
to test whether specific components of the model were effective, it cannot provide conclusive 
answers about how to modify the QOP model or how to create an entirely new program. 
Furthermore, as we have discussed, examining subgroup and site impacts does not provide clear 
guidance. Nevertheless, some of our implementation and impact findings are suggestive and 
provide some basis for speculation that might stimulate further research into programs designed to 
improve the educational and employment outcomes of at-risk youth. 

Start E artier. QOP was designed to start at the beginning of ninth grade. However, two 
factors prevented the demonstration sites from becoming operational and nearing the peaks of their 
learning curves until the middle of that school year or even the following year. First, obtaining from 
each school accurate enrollment hsts and information on students' eighth-grade performance (used 
for determining QOP ehgibihty) proved difficult and led to delays in determining eligibility, 
contacting students, and initiating program activities. Second, as discussed above, many program 
staff admitted that they did not fuUy understand the program model until at least a year into the 
demonstration. Thus, Q 0 P or another program might be more effective if it enrolled students and 
were fuUy operational at the beginning of ninth grade. Such an approach would reguire substantial 
investments in training and preparation before the start of the program, including, for example, 
summer training sessions for dl case managers and program staff and procedures for rapidly 
identifying ehgible students. 

However, starting at the beginning of ninth grade may not even be early enough. Programs 
intended to assist disadvantaged students may need to start earher, for example, in late elementary 
school or at the beginning of middle school. By the beginning of ninth grade, many youth face 
substantial barriers to academic success; they are already performing well below grade level and 
engaging in risky behaviors. As discussed above, QOP's developmental component was 
implemented more successfully than some other components of the program but was geared to 
preventing rather than remediating the effects of risky behaviors. Thus, intervening earher, as was 
suggested by QOP case managers, might help prevent youth from engaging in risky behaviors 
(Maxfield et al. 2003a). 

Recent research also reinforces the importance of early intervention, showing a strong 
relationship between performance early in high school and dropping out later. According to 
Hirschman, Pharris-Ciurej, and Whlhoft (2006, p. 15), "Failing a class in the first semester of high 
school or having a GPA below 1.0 (almost the same thing) resembles a death sentence in terms of 
high school graduation." To prevent this "death sentence," a program might need to change 
behaviors starting at the beginning of ninth grade or even earlier. Our results, which suggest that 
QOP was more effective for students age 14 or younger at the beginning of ninth grade, might 
provide some support for early intervention. The younger students had likely not been previously 
held back a year (or more) and, at the beginning of ninth grade, might have had fewer barriers to 
success as compared with students over age 14 when they enter high school. 

Individualize More. QOP was designed to provide comprehensive services to a broad set of 
enrollees. QOP enroUees in the demonstration included undocumented residents, youth in special 
education programs, youth with disabilities, teen parents, youth involved in the juvenile justice 
system, out-of-school youth, and youth who were one or more grades behind in basic skill levels 
(Maxfield et al. 2003a). Thus, their service needs were diverse. Many of the QOP services were to 
be provided to all enroUees whUe other services were to be made available to enrollees as needed. 
Q 0 P's case management component was designed to identify each enroUee's needs. However, with 
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the case management component not fully implemented, sites may not have tailored services 
effectively for each enrollee. 

While providing a breadth of services may be important, it may be egually important to target 
those services to each individual, according to individual needs. Thus, a program could offer a 
broad set of services, across aU enroHees, but with each enrollee receiving fewer services— those 
deemed critical for that enrollee's success. A program would need to place strong emphasis on 
carefully identifying each enrollee's needs. Maxfield et al. (2003a) suggested different program 
services for four different types of enrollees: those who had not yet engaged in risky behaviors; 
those who were held back in high school for one or more years but had not dropped out; those who 
had dropped out of high school and could not be persuaded to return; and those who had a child, 
were engaged in substance abuse, or had already engaged in criminal activity. For example, enreUees 
who had been held back in high school for one or more years but who did not drop out could 
receive services targeted to high school graduation, such as remedial academic services, while 
enroUees who had already dropped out of high school and could not be persuaded to return could 
receive services focused on receipt of a GED and enrollment in community college or other 
vocational training programs. 
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Appendix A 

Obtaining an Evaluation Sample and Conducting Random Assignment 
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Four steps led up to and concluded with random assignment: (1) developing lists of 
eligibles, (2) initial sampling, (3) obtaining consent, and (4) random assignment. These steps 
needed to be completed to obtain an evaluation sample for the QOP demonstration. 

To implement the four steps in the seven sites, we developed an individuahzed Student 
Selection Plan (SSP) for each site by customizing a generic plan to accommodate local 
circumstances. Exhibit A. 1 displays the generic plan. As it turned out, few accommodations 
to local circumstances were required; therefore, aU of the SSPs were similar. The main 
differences in the sites' SSPs concerned the number of QOP schools, how QOP slots were 
allocated across schools, and the dates of sampling and random assignment. In the three 
sites with more than one QOP school, the Q 0 P CBO was responsible for determining how 
slots would be allocated across the schools. 

Although random assignment was successfully implemented in the seven demonstration 
sites, the sites encountered three main problems in implementing the evaluation design: (1) 
developing accurate lists of eligibles, (2) contacting students, and (3) collecting completed 
forms. In the remainder of this appendix, we discuss these implementation problems in the 
context of the four steps hsted earlier. Although we present examples from individual sites, 
the examples usually illustrate experiences common to most or all sites. 



Developing Lists of Eligibles 



As shown in the model SSP, the generic instruction to each site was as follows: 

E adi school should compile a list of students who have entered the 9th grade for the first 
time in the current academic year and send the list to M PR. For arery student, the 
list should indude at least two pieces of identifying information and the students' 8th 
grade G PA . 

Fulfillment of this instruction completed the site's responsibihty. Then: 

For each school, MPR will rank students- from highest to lowest- according to their 
G PA s from the 8th grade. The students in the bottom two- thirds of the G PA 
distribution for their school are eligible. 

Although seemingly straightforward, these first two steps in implementing the 
evaluation design were probably the most difficult. They might also prove to be among the 
more difficult steps in implementing an ongoing QOP program. There were two main 
problems in developing an accurate hst of eligibles for a school: (1) determining current 
enrollment and (2) calculating G PAs. 
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Exhibit A. 1 



Quantum Opportunity Program Student Selection Plan 



This plan outlines the steps for selecting students for the Quantum Opportunity 
Program (Quantum). For each step, we have listed the responsibilities of local Quantum 
staff (including staff of the participating high schools) and the responsibilities of 
Mathematica Policy Research (MPR) staff. 

L Submitting L ists of Students. 

Quantum. Each school should compile a list of students who have entered the 9th 
grade for the first time in the current academic year and send the hst to MPR. For eveiy 
student the hst should include at least two pieces of identifying information and the 
student's 8th grade G PA. 

2. Identifying Eligible Students. 

MPR. For each school, MPR wih rank students— from highest to lowest— according 
to their grade point averages (G PAs) from the 8th grade. The students in the bottom two- 
thirds of the G PA distribution for their school are eligible for Quantum. 

3. Selecting a Group of Eligible Students to Receive Quantum Infoimation and 
Consent Packets. 

MPR. MPR will randomly select a group of 132 ehgible students from ABC High 
School and 88 eligible students from XYZ High School. MPR will send the list of selected 
students to Quantum staff on [date]. If permission is obtained from their parents, these 
students will be the study group. Only some (about half) of the students in the study group 
will later be selected, at random, to participate in the Quantum program. 

4. D istiibuting Q uantum Infoimation and C onsent Packets. 

Quantum. Quantum staff should distribute Quantum information and consent 
packets to all 220 students in the prospective study group. The packet will contain a cover 
letter from the student's school, a brochure describing the Quantum program and the 
Quantum study, a consent form seeking parental permission for the student to participate in 
the study, and a locator form. Quantum staff should make copies of the cover letter (on 
school letterhead) and copies of the consent and locator forms and assemble the packets. 

MPR. MPR will draft all materials for the Quantum information and consent packet. 
MPR wiU also make copies of the brochures and send these to Quantum staff. 



A-4 




Exhibit A.l (continued) 



5. C oUecting C ompleted C onsent and L ocator F orais. 

Quantum. Quantum staff should collect completed consent and locator forms. When 
a student returns completed forms, Quantum staff should attach preprinted labels for that 
student to the forms. It is important that completed consent and locator forms be obtained 
for all 132 students at ABC High School and 88 students at XYZ High School so that eveiy 
interested student will have an opportunity to be considered for participation in the 
Quantum program. Quantum staff will be responsible for purchasing an incentive item and 
distributing it to students who promptly return completed consent and locator forms. 

MPR. MPR win provide two preprinted labels for each student, one label for the 
consent form and one label for the locator form. MPR will pay for the incentive. 

6. Submitting C onsent and L ocator F oims. 

Quantum. Completed consent and locator forms should be sent to MPR at least 
weekly. 

7. Selecting Students for the Q uantum Program. 

MPR. MPR will compile a hst of all students for whom affirmative consent and a 
completed locator form have been obtained. The hst will be sent to Quantum staff for 
verification. 

Quantum. After verifying that the hst of students with affirmative consent and 
completed locator forms is correct. Quantum staff should sign the hst and send it to MPR. 

MPR. From the hst of students with affirmative consent and completed locator forms, 
MPR whl randomly select 60 students from ABC High School and 40 students from XYZ 
High School to participate in the Quantum program. Students who are not randomly 
selected for the Quantum program will be assigned to the control group for the study. On 
[date], MPR will send lists of Quantum group students and control group students to 
Quantum staff. 

Quantum. Quantum staff should notify all students about their group assignments 
(Quantum or control), and should inform MPR when ah students have been notified. After 
notifying Quantum students of their selection. Quantum staff should begin recruiting them 
for participation in the Quantum program. Only students randomly selected for the 
Quantum group may participate in the Quantum program. Students assigned to the control 
group and students who did not receive or did not complete consent and locator forms 
cannot participate in the Quantum program. All students in the Quantum and control 
groups are part of the Quantum study. 
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Exhibit A. 1 (continued) 



8. Submitting Lists of Quantum Participants. 

Quantum. To provide data for analyses of Quantum participation patterns, Quantum 
staff should send to MPR a list of all students participating in the Quantum program on the 
following dates: .... After [date], a hst of Quantum participants should be submitted eveiy 
twelve weeks. 



If this plan meets with your approval, please sign below and return to MPR. If you 
have any guestions concerning this plan or any other issues related to the study, please call 
[MPR site liaison] at [phone number]. Thank you for your assistance in developing this plan. 



Quantum Coordinator 



Date 
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Deteimining Cuirent Enrollment 



As a rule, most QOP schools did not know precisely which students were enrolled in 
the ninth grade. The explanation was sporadic attendance by many students combined with 
high turnover, both from year to year and within a year, as students' families moved 
frequently. 

Although we considered requesting first-day-of-school enrollment lists, we learned from 
school and district staff that such lists would be unreliable.^^ Many students expected to 
enroU in a school do not do so, and many unexpected students enroll. Moreover, some 
students do not attend school for the first few weeks of the year, especially if school starts 
before Labor D ay. 

In heu of a first-day-of-school enrollment hst, we accepted the first properly 
constructed list (with grades) that a school could produce. Such a list typically became 
available a month or more after school started. 

Even several weeks into the school year, however, students continued to transfer from 
school to school, and some students had attended classes on only a few days. For example, 
five weeks into the school year, one QOP school constructed a hst of ninth graders who 
were not repeating the ninth grade and were not inehgible because of a disabihty. The 
school constructed a second hst of such students two weeks later. Nearly one out of every 
six students on the first hst was not on the second hst. However, out of every five students 
dropped, one was replaced by a new student. We suspect that many of the students that 
were dropped had left the school before the construction of the first hst and that school 
record keeping was just catching up to student movements. Nevertheless, reports by school 
and Q 0 P staff suggested that some dropped students and some added students probably 
had moved during the two -week period between lists. 

Once a school had a hst of currently enrolled ninth graders, "categoricahy inehgible" 
students— students repeating the ninth grade and disabled students for whom QOP would 
have been inappropriate in the school's judgment— had to be dropped from the hst. 
Although a couple of schools neglected to drop a category of inehgible students in a first 
attempt to develop a hst of ehgibles, none of the schools in the demonstration appeared to 
have any significant difficulties in identifying categorically inehgible students. 



’3 Even if first-day-of-school lists had been more reliable, schools generally were not prepared to produce 
them because doing so would have Interfered with regular school activities. 

If an ongoing QOP program were to start delivering services very near the beginning of the school 
year, the proportion of students selected for Q 0 P who turned out to have transferred to other schools would 
be much higher than in this demonstration, in which service delivery started almost half way through ninth 
grade. Also, many (If not most) students new to the school district or coming from middle schools within the 
district that are not traditionally feeder schools for the Q 0 P high school would effectively be Ineligible for 
QOP. As we discuss later, however, even when lists are constructed several weeks into the school year, many 
students new to a district are ineligible for Q 0 P because no grades are available for them. 
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Calculating GPAs 



After developing a list of currently enrolled ninth graders and dropping from the hst 
categorically inehgible students, a school attempted to calculate an eighth-grade G PA for 
each remaining student on the hst. Initial conversations with school staff revealed confusion 
about what would constitute an acceptable G PA. Some thought that G PA means a credit- 
weighted average on a four-point scale. We were told, for example, that it would not be 
possible to obtain G PAs for one school because only "grade averages" (on a 100-point scale) 
were available. Such confusion was easily ehminated by distributing a brief memorandum 
discussing the calculation of G PAs and other issues pertaining to ehgibility.’^ 

Although it might be more serious if QOP were a permanent rather than a 
demonstration program, another minor problem was that two schools did not have the 
resources to calculate GPAs. For one school, QOP staff calculated GPAs from students' 
eighth-grade transcripts. For the other school, we performed the necessary calculations.’® 

The most serious problem that arose in attempting to calculate GPAs was obtaining 
eighth- grade transcripts for students who were new to the local public school system after 
transferring from other school systems or private schools. The typical procedure for 
calculating G PAs involved two steps. First, district and school staff obtained G PAs for as 
many students as possible from a computerized database. That database rarely included 
grades for students new to the system. Second, if the database contained no grades, QOP 
staff searched a student's paper files for an eighth-grade transcript. If a transcript were 
available, QOP staff calculated a GPA by hand.” More often than not, however, no 
transcript appeared in a student's file. 

For one QOP school, no grades were available in the district's database for nearly 17 
percent of students. QOP staff were able to locate an eighth-grade transcript for only 20 
percent of those students. So, overall, GPAs could be calculated for just 87 percent of the 
school's categorically ehgible students.” 



’5 Some school staff were also confused about how to rank students based on grades. 0 ne school initially 
had a separate ranking for each middle school that fed students to the high school. We eliminated the 
confusion by having each QOP school send us a list with names and grades for aU categorically eligible 
students. Then, we ranked students and identified the (fully) eligible students, that is, the students in the 
bottom two-thirds of the grade distribution. 

’6 For another school, QOP staff entered G PAs from students' transcripts into a database. 

11 The main difficulty in this case was making sure that the G PA was comparable to other students' 
G PAs— that it was, for example, on the same scale. 

In two other schools, G PAs could be calculated for 88 and 65 percent of categorically eligible students. 
For the first school, QOP staff had to track down G PAs for about one in six students for whom G PAs could 
be calculated. For the other school, it was two in five. 
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The consequences of this problem were borne by students. Because there was no basis 
for ranking students for whom a G PA could not be calculated, such students were inehgible 
for Q 0 P. Thus, potentially many students who were experiencing the difficulties of entering 
a new school system had no opportunity to enroU in Q 0 P because their transcripts were less 
mobile then they were. 

While problems arose in determining enrollment and calculating G PAs, we should note 
that in some sites accomplishing both of those tasks seemed more than twice as difficult as 
accomplishing either one of them. The problem was that information in a school system 
was dispersed. The Q 0 P school had more accurate enrollment data than the central district 
office but much less easy access (if any access) to computerized records of grades.™ 
Moreover, there was rarely one person who had a good working knowledge of each data 
source. This problem was made worse by the fact that the most knowledgeable person 
generally did not have the authority to take direction from a third party (us or the CBO ) or 
to make judgments such as whether a particular special education student should be ehgible 
for Q 0 P. Yet another obstacle was that schools often had little experience in responding to 
information requests such as those that we made. A final obstacle was that despite 
enthusiasm for QOP and a cooperative spirit on the part of school and district staff, 
determining which students were eligible for Q 0 P was generally not a high priority. Thus, 
the resources needed to do the job accurately were not always available.*^® 

We discovered many errors in some lists submitted to us and returned the lists to the 
schools for corrections.’^^ Nevertheless, because httle information was available for assessing 
the accuracy of the lists, we are certain that the final hsts contained errors, some of which 
were discovered later in the process of obtaining an evaluation sample. Only by requesting 
more data and further burdening the schools could the numbers of errors have been 
determined and reduced substantially. 



™ For one QOP school, an enrollment list prepared by the central district office missed three-fifths of the 
students on the school's own enrollment list. At the same time, over one-quarter of the students on the 
district-prepared list were no longer enrolled according to the school's list. For another school, the differences 
were less extreme, but still large. The district's list missed one-quarter of the students on the school's list, while 
about one-sixth of the students on the district's list were not on the school's list. 

so For school staff, the highest priority was running the school. When attention was given to QOP, the 
highest priority of school and QOP staff was, understandably, serving students. Promoting fairness by 
ensuring the accuracy of the list of eligible students, most of whom would not be served by QOP, was a lower 
priority. 

SI The most common errors were excluding students new to the school system and including repeaters. 
On hsts submitted by one site, for example, we discovered that new students had been excluded. We 
discovered this by observing that not a single student had attended eighth grade in a school outside of the city. 
For one school, which had grades 9 through 12, we noticed that several students had attended that school the 
previous year, suggesting that repeaters had been included. 
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Initial Sampling 



In all but two Q 0 P schools, we drew a simple random sample (without replacement) 
from the list of students eligible for Q OP. The selected students were ehgible for random 
assignment if consent was obtained for them to participate in the evaluation. The students 
who were not selected for the initial sample were not eligible for random assignment and 
therefore no longer had an opportunity to participate in QOP. We did not draw random 
samples for two schools because the number of eligible students was less than the target 
sample size. We conducted sampling independently for each school. 

The initial samphng of eligibles had two purposes: (1) to minimize the impact of the 
evaluation on students and (2) to minimize the burden on QOP staff. Although such 
concerns about impact and burden arise in every random assignment evaluation, they were 
heightened in the Q 0 P demonstration because in several of the Q 0 P schools, the number 
of eligible students was substantially greater than the target size of the evaluation sample 
(100 in the Ford- funded sites and 200 in all but one of the DOL-funded sites). Thus, there 
were many extra students who would not be selected for the limited number of Q 0 P slots 
(50 in the Ford-funded sites and 100 in aU but one of the D 0 L-funded sites) and were not 
needed to form a control group for the evaluation. Locating those students, telhng them and 
their parents about Q 0 P and the evaluation, and obtaining consent for them to participate 
in the evaluation would have substantially increased the workload of Q 0 P staff. Moreover, 
many more students than necessary would have had their hopes raised, only to be 
disappointed later. Samphng hmited the number of disappointed students. 

Once we decided to sample eligible students, we had to determine the size of the 
sample. We wanted to obtain a control group for each school that was the same size as the 
QOP group, implying a target sample size that was twice the number of available QOP slots. 
However, if we had drawn a sample with as many students as the target size of the 
evaluation sample, we would have had no surplus to ahow for students who left the QOP 
school between development of the school enrollment roster and samphng (because they 
transferred, dropped out, or were expelled); for students who simply could not be located 
(or, if located, could not be contacted); and for students for whom consent was denied 
(explicitly or, by nonresponse, implicitly). Losing those students and dropping below the 
target size for the evaluation sample because we had no surplus would have reduced the 
precision of impact estimates. On the other hand, if we had a generous surplus, we would 
have disappointed more students than necessary and excessively burdened QOP staff. 

After weighing these considerations, we drew for each school a sample of ehgible 
students that was 10 percent larger than the target size for the evaluation sample. 
Accordingly, if a CBO in a DOL-funded site with two QOP schools specified that one 
school would have 60 QOP slots while the other would have 40, we drew a sample of 132 
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(= 60 H 2 H 1.1) students for the first school and 88 (= 40 H 2 H 1.1) students for the second 
school.*^^ There were two exceptions to this rule for setting the sample size. First, if (the 
number of QOP slots H 2 H 1.1) was greater than the number of ehgible students in a 
school, we selected all of the eligible students. Second, last-minute changes in the allocation 
of QOP slots across the Memphis schools caused minor deviations from the formula.*^^ 

Our sample size choice was a compromise between the ideal of randomly selecting a 
sample of ehgible students and getting consent for every one of them and the reahty that it 
could not be done. T o emphasize the importance of reaching out to every ehgible student- 
regardless of the student's initial interest in QOP— as a fundaments principle of the 
program, we instructed QOP staff to make every reasonable effort to obtmn a completed 
consent form for each student in the sample. In addition, as we discuss later, we imposed 
safeguards to ensure that such efforts were undertaken. 



Obtaining Consent 



After selection of the initiS sample for each school, QOP staff attempted to distribute 
information packets to each selected student. The packets contained a cover letter from the 
student's school (usually signed by the principal), a brochure describing the program and the 
evaluation, and a parental consent form for the evaluation.*^^ In addition to cohecting 
completed consent forms, QOP staff were responsible for having students and their parents 
complete a "locator" form that would provide tracking information to enable us to contact 
students for fohow-up data cohection. Ah but one site chose to include the locator form in 
the packet with the other materials. 

Although sites varied in how they distributed and cohected completed consent and 
locator forms, a typical approach involved the fohowing four steps: (1) hold an in-school 
assembly to speak with students and distribute packets; (2) tiy to find at the school the 
students who did not attend the assembly; (3) reguest that students return completed forms 
to a specified location (usually an office in the school); and (4) follow up with telephone calls 
and, more often, home visits to meet with parents and obtain completed forms. QOP staff 
carried out these steps, sometimes with hmited assistance from school staff.*^^ 



The factor of 2 in the mathematical expressions reflects the fact that we wanted to obtain a control 
group that was the same size as the Q 0 P group. The QOP group had as many students as there were QOP 
slots. 



To avoid any further delays in enrolling students in Q 0 P, we did not draw a supplemental sample If the 
10 percent surplus in the original sample turned out to be too small. Instead, we allowed the control group to 
be smaller than the Q 0 P group. 

Spanish language materials were available. 

The Y akima site deviated most dramatically from the approach outlined. Confidentiality restrictions 
severely limited the role of QOP staff until parental consent was obtained. Therefore, school staff were 
responsible for locating students, distributing materials, and collecting completed consent forms. 
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The home visits were especially important in obtaining completed forms from as many 
students in the sample as possible. First, a home visit was the first contact with a nontrivM 
fraction of students who attended school sporadically. Second, it was often the most rehable 
means of getting forms dehvered to a parent, completed, and returned to Q 0 P staff. 

To expedite the process of obtaining consent, all sites offered a nominal incentive, such 
as movie theater or grocery store gift certificates, for returning completed forms promptly. 
Nevertheless, obtaining consent was difficult and time-consuming. For the median student, 
one month elapsed between the time when the student was selected for the sample of 
students who could receive information packets and the time when we received a completed 
consent form for that student.*^® For 17 percent of students, more than seven weeks elapsed. 

Two of the implementation problems mentioned earlier arose in the process of 
obtaining consent and explain why die process was so difficult and time-consuming. These 
problems were (1) contacting students and (2) collecting completed forms. 

Every site encountered difficulties in locating and contacting a substantial fraction of 
students. The main reason was that the students' famihes moved freguently, which was an 
explanation noted earlier for why schools had trouble in determining their current 
enrollment. For some of these students, Q 0 P staff learned that after the school constructed 
the enrollment hst used for identifying ehgibles and drawing the initial sample, the students 
guit attending the QOP school, often because they had moved and transferred to another 
school. For students stiU thought to be hving nearby and enrolled in the Q 0 P school, QOP 
staff often discovered that the contact information contained in school records was badly 
out of date. Sometimes, the information was current but inaccurate, referring, for example, 
to a nonexistent address. Using various means, such as talking with a student's friends, 
QOP staff were often able to determine where a student lived. However, it was still difficult 
to contact some students' families because there was no telephone in the home, no adult was 
at home much of the time, or a convenient meeting time could not be arranged. 

Problems did not end when contact was made with a student. An information packet 
given to a student often was not dehvered to the student's parents, and sometimes 
completed forms were not returned to school. In other instances, parents did not read the 
materials or complete the forms. Sometimes, the seeming lack of reliability was attributable, 
in fact, to an initial lack of interest in QOP, concern about the time commitment reguired, 
or suspicions about government programs. QOP staff discussed these issues at length with 
students and parents. To address concerns about time commitments, for example, QOP 
staff explained that students were not obligated to participate in Q 0 P if selected and could 
refuse to answer survey guestions or take evaluation achievement tests. 



86 This figure overstates the time required for a site to obtain a compieted form. First, a day or two— 
somehmes more— eiapsed between sampie seiection and the first attempt to contact a student. Second, a site 
typicaiiy waited untii it had received compieted forms for severai students before shipping the forms to us. 
Therefore, some forms may have been in a site's possession for a few days before being shipped. Even 
considering these two factors and the time required for shipping, we figure that it took, on average, two to 
three weeks to contact a student and coiiect a compieted consent form. 
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G enerally, when less intrusive approaches had failed in getting forms completed, the 
most effective strategy seemed to be for Q 0 P staff to visit parents in the students' homes 
and wait there while the parents completed the forms. In contrast, telephone calls to parents 
achieved only hmited success when previous contact with the student alone had failed. 

The only other problem in obtaining completed forms pertained to how they were 
completed— specifically, ensuring that the consent form was properly marked and signed 
and that the most important items on the locator form were provided. Although about 40 
percent of locator forms (and 1 percent of consent forms) had deficiencies and were 
returned to sites, correcting the deficiencies was usually straightforward and caused only 
minor delays in random assignment. 



Random Assignment 



After three to four months of developing a list of eligibles and obtaining completed 
consent and locator forms for students, the final activities reguired to complete random 
assignment took about one day. The main activities were a series of checks designed to 
ensure that random assignment was conducted properly and fairly. 

To be eligible for random assignment, a student had to (1) be eligible for QOP (some 
students were found to be ineligible after selection of a school's initial sample), (2) have 
parental permission to participate in the evaluation, and (3) have a completed locator form. 
Before we proceeded to random assignment of the ehgible students to QOP and control 
groups, we reguired QOP coordinators to: 

• V erify that the list of students eligible for random assignment was accurate. 

• Verify the planned allocation of QOP slots across schools (if there was more 
than one school). 

• V erify that QOP staff had made good-faith efforts to locate, contact, and obtain 
completed forms for students who were not eligible for random assignment. 

Typically, the last verification involved a student- by-student review of the actions taken 
by QOP sttff and the outcome (e.g., QOP staff discovered that the student moved to 
another state three months earher). Sites had to establish that parental permission and a 
completed locator form were highly unhkely to be forthcoming in the near future. 

After the verifications were completed, we randomly assigned students ehgible for 
random assignment to QOP and control groups. One student was assigned to each available 
QOP slot regardless of how many students were eligible for random assignment. We 
conducted random assignment independently for each school. 
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After completing random assignment for a site, we sent the Q 0 P coordinator the list of 
Q 0 P group students and the hst of control group students. Q 0 P staff were responsible for 
notifying all students about the outcome of random assignment. T o maintain the integrity of 
random assignment, we imposed two rules: (1) a student in the control group could not 
participate in Q 0 P and (2) a student who was not ehgible for random assignment could not 
participate in QOP. To our knowledge based on several monitoring activities, these rules 
were not violated. 



SCHOOL-BY-SCHOOL SUMMARY OF SAMPLE DEVELOPMENT 



Table A.l shows how the evaluation sample was developed for each school. The first 
row shows the number of slots allocated to each school. The second row in the table- 
headed "G PA Eligibles"— shows the number of students in each school who were attending 
the school, were entering ninth grade for the first time, were appropriate for QOP in 
accordance with apphcable laws and regulations, and were in the bottom two -thirds of the 
grade distribution based on grades from the eighth grade (among students satisfying the first 
three criteria). The number of eligible students ranged from 82 to 523 across the QOP 
schools. Using the procedures described in detail earlier, we selected from the hst of G PA 
Eligibles an "Initial Sample" consisting of the number of students shown in the third row. 
Then, we instructed QOP staff to obtain consent for participation in the evaluation for ah 
students in the initial sample. 

As discussed in the main text, about five percent of the students in the initial sample— 
the students in the row headed "Ineligibles"— were determined to be ineligible for QOP 
based, in most instances, on evidence from school records indicating that a student had 
never attended the Q 0 P school or had left the school early in the school year before QOP 
eligibility was determined. The parents/ guardians of about another seven percent of the 
students in the initial sample never responded to Q 0 P staffs attempts to obtain consent. 
As we noted before, there was strongly suggestive evidence from school staff or other 
sources— but not definitive evidence from school records— that many of these students 
were, in fact, inehgible. However, in some instances, the failure to respond probably was a 
passive denial of consent. Parents/ guardians actively denied consent for another two 
percent of the initial QOP sample. Before we would conduct random assignment for a 
school, QOP staff had to verify that they had made substantial efforts to contact and obtain 
consent from the nonrespondents.*^^ 

The "Consenters" row in Table A.l gives the number of students who were eligible for 
random assignment and therefore constitute our evaluation sample. Prom among these 
students, we filled the available QOP slots independently for each school by simple random 



87 The nonresponse and active denial of consent percentages are the same when the base for the 
percentages is the number of students in the "N et Eligible Sample" rather than the initial sample. 
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sampling without replacement. Students who were selected for QOP became QOP 
enroHees. Students who were not selected for Q 0 P became the control group.’^’^ 



One seemingly minor limitation of the group of consenters as a representative sample of the population 
of students who satisfy the Q 0 P eligibility criteria is that a few implicit and explicit denials of consent might 
not have occurred in the absence of the evaluation. However, it seems unlikely tiiat more than a trivial number 
of students would have accepted a 100 percent chance to participate in QOP but rejected a 50 percent chance 
that was essentially costless. 
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Table A.1. Development of the Evaluation Sample 
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Appendix B 
The Baseline Data 
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Baseline data represent sample members' characteristics that were unaffected by QOP, 
either because they were determined prior to the demonstration or because— hke age— they 
cannot be affected by a social program. We used basehne characteristics to: 

• Correct for nonresponse bias in the impact estimates. 

• Adjust for random differences between the Q 0 P group and the control group. 

• E stimate impacts on subgroups of enrollees. 



Data Sources for the Baseline Database 



The baseline database contains information on sex, date of birth, race, ethnicity 
(Hispanic origin), and eighth-grade grade point average (G PA). Because D 0 L elected not to 
conduct a basehne survey, data on these characteristics were cohected from four other 
sources: (1) the database used to determine ehgibihty for QOP; (2) the telephone survey 
administered during the fall and winter of the fifth year after sample members entered the 
ninth grade; (3) high school transcripts; and (4) QO P case managers. The eligibility database 
included eighth-grade G PA and the name of the school attended at the beginning of ninth 
grade. It also often included date of birth, and for some schools, it included sex, race, or 
ethnicity. 



Development of the Baseline Database 



To develop the basehne database, we used the four data sources hierarchically in the 
order hsted above. If a value needed for the basehne database was available from the 
ehgibihty database, no other sources were consulted. Thus, the final source, QOP case 
managers, was used only when the value was not available from the first three sources. After 
using the first three data sources and imputing sex based on sample members' first names 
(for about 17 percent of sample members), there were no missing values for G PA in eighth 
grade, school attended, or sex, and there were only five missing values for date of birth. The 
missing data rates for ethnicity and race were 15 and 26 percent, respectively. We imputed 
for the missing values using a seguential hot deck procedure, which is described in Schirm et 
al. (2003). 
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Follow-Up Data From the Third Telephone Survey 
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Data on nearly all outcomes considered in our analysis of initial post- intervention 
impacts were obtained from our third telephone survey, which was administered more than 
nine years after sample members entered the ninth grade (more than four years after the end 
of the demonstration).®'^ The only outcomes based partly on other data are our measures of 
high school completion, which were constructed— as described in Appendix F— using data 
from the first, second, and third telephone surveys and high school transcripts. The first 
telephone survey was conducted during the fifth year of the demonstration and the second 
telephone survey was conducted two years after the end of the demonstration— more than 
two years before the third telephone survey. 

In this appendix, we describe the fielding procedures for the third telephone survey.®® 
Then, after discussing the response rates to the third telephone survey, we examine the 
prevalence of missing values for outcomes, that is, item nonresponse. 



Fielding Procedures for The Third Telephone Survey 



Table C.l hsts the sites and schools that participated in the QOP demonstration. Table 
C.2 presents the dates for all follow-up data collection activities completed to date. 

Initial interviews for the third telephone survey were conducted using computer- assisted 
telephone interviewing (CATI) followed by in-person follow-up of nonrespondents. The 
interview took about 20 minutes to complete. A copy of the questionnaire is available upon 
request. 

Each sample member was mailed a letter prior to the start of interviewing. The letter 
indicated that we would call for an important foUow-up study and encouraged the sample 
member to participate. In addition, the letter indicated that we would pay $25 for 
completing the interview. 

Overall, 64 percent of the sample members who responded did so via telephone, 35 
percent responded in-person, and 1 percent responded by mah.®^ 



The program in the Washington, D C site started one year later than the programs in the other sites, and 
in that site, die third telephone survey was conducted more than eight years after sample members entered the 
ninth grade (more than three years after the end of the demonstration). 

90 The fielding procedures for the previous data collection activities are presented in Schirm et al. (2003) 
and Schirm and Rodiiguez-Planas (2004). 

91 All of the in-person completes were obtained by having the sample member call one of our telephone 
interviewers using a cell phone provided by a field locator. The interview was then conducted using the CATI 
system and in the presence of the field locator (hence the designation "in-person"). 
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Response Rates for the Third Telephone Survey 



Table C.3 displays the completion rates for the third telephone survey and prior data 
collection activities relative to all 1,069 sample members in the original sample. The figures 
are presented separately for Q 0 P and control group members and are presented for the full 
sample and by school. Table C.4 shows survey response rates for each data collection 
activity, where for each activity, the rates exclude those sample members who were knovm 
to have died prior to the start of that data collection activity or who had been removed from 
the tracking database prior to the data collection activity. 

According to Table C.5, which provides more detail on the survey response rates and 
the final disposition report for the third telephone survey, the overall response rate to the 
survey was 76 percent. The difference in response rates between the QOP and control 
groups was 3 percentage points overall (77 percent for the QOP group and 74 percent for 
the control group). As with the previous surveys, the differential varied across schools and 
sites. Across sites, the largest difference in response rates between the QOP and control 
groups was for the Washington, D C, site (12 percentage points). As indicated in Table C.5, 
most nonrespondents to the third telephone survey were sample members who could not be 
located. 



Missing Values 



Item nonresponse was uncommon— often less than one percent— for most outcome 
measures used in the impact analysis (see Tables C.6 and C.7).®^®^ Moreover, item 
nonresponse differed very little between the Q 0 P and control groups. 



92 At the beginning of the data collection period, a small number of employment questions were 
mistakenly skipped. We attempted to recontact about 500 affected sample members, and successfully reached 
92 percent (93 percent of the QOP group and 90 percent of the control group). Item response rates listed in 
Tables C.6 and C.7 combine results of the original and recontact surveys. 

93 The lowest response rates are for outcomes pertaining to employment and earnings during the 12 
months prior to the survey. Some respondents did not know their total earnings or number of weeks worked 
during the 12-month period. N onresponse to the recontact survey also contributed to the lower response rates 
for these outcomes. 
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Table C.1. OOP Sites and Schools 



QOP Site 


Schools 


Fort Worth, TX 


Paschal High School 


Cleveland, OH 


Collinwood High School 


Washington, DC 


Anacostia High School 
Eastern High School 


Houston, TX 


Austin High School 
Yates High School 


Memphis, TN 


Carver High School 
Hamilton High School 
Hillcrest High School 


Philadelphia, PA 


Ben Franklin High School 


Yakima, WA 


Davis High School 


Table C.2. Data Collection Fielding Dates 


Instrument 


Fielding Dates 


Non-DC In-Person Survey/Achievement Tests 


February - April 1999 


DC In-Person Survey/Achievement Tests 


April 2000 


Non-DC First Telephone Survey 


November 1999 - June 2000 


DC First Telephone Survey 


November 2000 - April 2001 


Non-DC School Records 


September 1999 - December 2000 


DC School Records 


December 2000 - April 2001 


Second Telephone Survey 


September 2002 - April 2003 


Third Telephone Survey 


January 2005 - September 2005 
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Source: In-person survey, achievement tests, telephone surveys, and transcripts. 




Table C.4. Response Rates for Data Collection Activities (Percentages) 
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Table C.5. Third Telephone Survey Dispositions, by Site and QOP/Controi Status 
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end of field period (3%) (3%) (4%) (0%) (0%) (0%) (3%) (4%) (1%) (0%) (0%) (0%) 

Deceased 000220 532 2 0 2 

Ineligible 110000 000 0 0 0 




Table C.5 (continued) 
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Percentages exclude 19 deceased cases and 1 ineligible case. 




Table C.6. Item Response Rates for Outcomes Pertaining to High School Completion, 
Postsecondary Education and Training, and Employment and Earnings (Percentages) 







QOP 


Control 


Total 


Outcome 




Group 


Group 


Sample 


Received HS diploma 




100.0 


100.0 


100.0 


Received HS diploma or GED 




100.0 


100.0 


100.0 


Ever attended or currently attending a 4-year college 




99.8 


99.7 


99.7 


Completed at least 1 year at a 4-year college 




99.3 


99.7 


99.5 


Completed at least 2 years at a 4-year college 




99.3 


99.7 


99.5 


Earned a bachelor’s degree 




99.8 


99.7 


99.7 


Ever attended or currently attending a 2- or 4-year college 




99.8 


99.7 


99.7 


Completed at least 1 year at a 2- or 4-year college 




99.3 


99.2 


99.2 


Completed at least 2 years at a 2- or 4-year college 




99.3 


99.2 


99.2 


Earned a bachelor’s or associate’s degree 

Ever or currently in college, voc/tech school, apprenticeship, 


or 


99.8 


99.7 


99.7 


military 




100.0 


99.7 


99.9 


Completed 2 years of college or military service, completed voc/tech 








school or an apprenticeship, or honorably discharged from 
military 


the 


99.3 


98.6 


99.0 


Completed an associate’s or bachelor’s degree, voc/tech school or 
an apprenticeship, in military for more than 2 years, or honorably 








discharged from the military 




99.8 


99.2 


99.5 


Ever or currently in college, voc/tech school, an apprenticeship, the 








military, or Job Corps 




100.0 


99.7 


99.9 


Currently in a 4-year college 




99.8 


99.7 


99.7 


Currently in a 2- or 4-year college 

Currently in college, voc/tech school, an apprenticeship, or 


the 


99.8 


99.7 


99.7 


military 




99.8 


99.2 


99.5 


Currently employed 




100.0 


99.2 


99.6 


Currently unemployed 




99.8 


98.9 


99.4 


Currently out of labor force 

Currently employed or in college, voc/tech school. 


an 


99.8 


98.9 


99.4 


apprenticeship, or the military 




99.8 


98.9 


99.4 


Ever employed 




100.0 


99.2 


99.6 


Employed in past 12 months 




99.8 


99.2 


99.5 


Percentage of weeks employed in past 12 months 




89.5 


87.6 


88.6 


Number of jobs in past 1 2 months 




92.7 


91.5 


92.2 


Tenure at current job 




99.8 


98.6 


99.2 


Usual number of hours worked per week in all current jobs 




99.5 


98.9 


99.2 


Works at least 35 hours per week at main current job 




100.0 


99.2 


99.6 


Total earnings in past 12 months 




79.7 


79.9 


79.8 


Hourly earnings at main current job 




96.1 


94.9 


95.6 


Has a job with health insurance 




99.3 


98.3 


98.9 


Has a job with paid time off 




99.5 


98.6 


99.1 


Has a job with pension or retirement benefits 




98.6 


97.7 


98.2 



Source: 



Telephone surveys and transcripts for the two outcomes pertaining to high school completion. 
Third telephone survey for all other outcomes. 
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Table C.7. Item Response Rates for Outcomes Pertaining to Risky Behaviors, Physical and 
Mental Well-Being, and Family Life (Percentages) 



Outcome 


QOP 

Group 


Control 

Group 


Total 

Sample 


Smoked cigarettes or used tobacco in past month 


99.5 


98.9 


99.2 


Smoked cigarettes or used tobacco daily in past month 


99.3 


98.9 


99.1 


Binge drinking in past month 


99.5 


98.9 


99.2 


Binge drinking on 8 or more days in past month 


99.5 


98.9 


99.2 


Used an illegal drug in past month 


99.5 


98.6 


99.1 


Committed a crime in past 3 months 


99.5 


98.9 


99.2 


Committed a crime in past 2 years 


99.5 


98.9 


99.2 


Arrested or charged in past 2 years 


99.5 


98.9 


99.2 


Convicted or pled guilty in past 2 years 

Served time in jail, prison, or detention home in past 2 


99.5 


98.9 


99.2 


years 


99.5 


98.9 


99.2 


Self-reported health is fair, poor, or very poor 
Physical or mental condition limited activities quite a lot or 


99.5 


99.2 


99.4 


could not work because of these limitations 


99.5 


99.2 


99.4 


Had first child before age 18 


98.6 


99.2 


98.9 


Currently living with natural children, but no spouse 


99.5 


98.6 


99.1 


Have children with whom not currently living 
Have child with whom not living and not providing any 


99.5 


98.9 


99.2 


regular child support 


99.5 


98.9 


99.2 


Currently receiving welfare 


99.5 


98.9 


99.2 


Currently receiving food stamps 


99.5 


98.9 


99.2 


Currently receiving welfare or food stamps 


99.5 


98.9 


99.2 



Source: Third telephone survey. 
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Appendix D 

OUTCOMESAND SUBGROUPS 
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This appendix describes the outcomes and subgroups of enroUees for which we 
estimated impacts. 



Outcomes 



The outcomes fall into four broad categories: 

5. H i^ School C ompMion. The outcomes in this category measure receipt of a high 
school diploma or receipt of a general educational development (GED) 
certificate. 

6. Postsecondary Education or Training. The outcomes in this category measure 
engagement in postsecondary education or training through college, 
vocational/ technical schools, certified apprenticeship programs, and the armed 
forces. 

7. E mployment and E amings. The outcomes in this category measure employment 
status, percentage of weeks employed, number of jobs held, job tenure, number 
of hours worked, total earnings, hourly earnings, and availability of job benefits. 

8. Risky Behaviors, Physical and Mental Well-Being and Eamily Life. The outcomes in 
this category measure substance abuse, criminal activity, involvement with the 
criminal justice system, health status, teen pregnancy, single parenting, 
noncustodial parenting, chUd support provision, and welfare receipt. 

Table D .1 displays the complete hst of outcomes by category. Most of the outcomes are 
self-explanatory, although severi require additional explanation, which is presented below. 

High School Completion 

Data on high school completion were obtained from the first, second, and third 
telephone surveys and from transcripts. Appendix F explains how we measured high school 
completion using these data.^^ 



94 All other outcomes considered in this report were measured using data from the third telephone survey 

only. 
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Risky Behaviors 

Substance Abuse. "Binge" drinking means drinking five or more drinks in a row. In 
the main text of this report, binge drinking is described as "freguent" if it occurs on at least 8 
out of the past 30 days. The outcome "used any drug in the past 30 days" indicates that the 
respondent reported using at least one of the following illegal drugs or types of illegal drugs: 
marijuana or hashish; cocaine or crack cocaine; heroin, opium, or methadone; stimulants; 
depressants; inhalants; or hallucinogens. Because the rates at which sample members were 
using most of the individual drugs were low and the evaluation samples for schools and sites 
were small, impacts for individual drugs could not be rehably estimated and are not 
presented. 

Criminal Activity. The outcomes "committed any crime in the past 3 months" and 
"committed a crime in the past 2 years" indicate that the respondent reported committing at 
least one of the following six crimes: (1) sold illegal drugs, (2) stole a motor vehicle, (3) stole 
something other than a motor vehicle, (4) attacked and seriously hurt or killed someone, (5) 
carried a hand gun, and (6) committed a sexual assault. Because the rates at which sample 
members were committing most of the individual crimes were low and the evaluation 
samples for schools and sites were small, impacts for individual crimes could not be rehably 
estimated and are not presented. 



Subgroups 



We present impacts for subgroups defined by basehne characteristics— sex, age, and 
G PA. T ^le D .2 hsts the subgroups and their sample sizes. 

When assessing impacts for the subgroups defined by rank in the basehne grade 
distribution, it is important to remember that to be eligible for QOP, a youth had to be in 
the bottom two -thirds of the grade distribution based on grades from the eighth grade. 
Thus, youth in the bottom third of the basehne grade distribution for QOP ehglbles were at 
or below the 22“'^ percentile in the distribution for ah youth, including those who were not 
eligible for Q 0 P based on their grades. Likewise, the youth in the middle and top thirds of 
the basehne grade distribution for QO P ehgibles were between the 22"'^ and 44*^ percentiles 
and between the 44*^ and 66*^ percentiles, respectively, in the grade distribution for all youth. 
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Table D.1. Outcomes 
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Employed in past 12 months 

Percentage of weeks employed in past 12 months 

Number of jobs in past 1 2 months 

Tenure at current job 
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Table D.2. Subgroups 

Subgroup 



Sample Size 



Sex 

Males 576 

Females 493 



Age when entered ninth grade 

14 or younger 706 

Over 14 363 

Rank in baseline grade distribution (based on eighth-grade GPA) 

In the bottom third of the grade distribution 380 

In the middle third of the grade distribution 359 

In the top third of the grade distribution 330 
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Appendix E 

Weighting, Impact Estimation, and Variance Estimation 
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Weighting to Adjust for Nonresponse 



We developed person-level weights to adjust for the potential effects of unit 
nonresponse. Unit nonresponse occurred, for example, when a sample member did not 
respond at all to the third telephone survey, that is, the sample member did not answer any 
questions on the survey.®^ About 24 percent of sample members did not respond to the 
third telephone survey. The unit nonresponse rate for the control group was higher than the 
overall rate, while the unit nonresponse rate for Q 0 P enrollees was lower than the overall 
rate. The difference between the unit nonresponse rates for the two groups was about 3 
percentage points. 

Differences in basehne characteristics between respondents and nonrespondents could 
potentially cause differences between the outcomes of respondents and those of 
nonrespondents. In such a circumstance, an impact estimated using data from respondents 
only (since there are no outcome data from nonrespondents) would be a biased estimate of 
the impact that we seek, which is the impact on all sample members, respondents and 
nonrespondents. The size of the bias is not estimable. 

To adjust for the effects of nonresponse and reduce potential nonresponse bias, we 
assigned weights to respondents. We assigned larger weights to the respondents who more 
closely resembled the nonrespondents in terms of basehne and other characteristics and 
smaher weights to the respondents who less closely resembled the nonrespondents.®® 
Although differential weighting of respondents tended to increase the variances of impact 
estimates (by measurable amounts), we accepted small increases in variances to enhance our 
confidence that we controlled nonresponse bias to the extent possible. 

Weights 

We derived two different sets of weights to adjust for nonresponse: (1) weights for the 
outcomes measuring high school completion; and (2) weights for all of the other outcomes. 

To derive the second set of weights, we estimated two separate logit regression models 
to predict the probability that each sample member responded to the third telephone 
survey— one for the Q 0 P group and one for the control group. T o derive the other set of 
weights, we repeated this process, defining a sample member as a "respondent" if we were 
able to ascertain whether the sample member received a diploma or GED.®’ Then, we 
estimated the impact on an outcome using the appropriate set of weights. 



95 In contrast, item nonresponse occurred when a sample member did not provide a valid answer to a 
question that was asked even tiiough he or she answered other questions on the survey. As shown in 
Appendix C, item nonresponse rates were typically very low. 

96 As described in detail below, we evaluated resemblance using response propensity scores. 

9’ As described in detail in Appendix F, we determined a sample member's high school completion status 
based primarily on that person's responses to questions asked during the first, second, and third telephone 

(rontinued) 
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We derived each set of weights by canying out the following four steps: 

1. We estimated a "best" logit model for predicting response propensity scores 
(probabilities) separately for QOP and control group members. The best 
regression model included 22 predictors that we "forced" into the model and 
additional predictors that we selected using an automated forward selection 
procedure with a liberal inclusion criterion.®® The predictors forced into the model 
were an intercept 10 school indicators, an indicator for whether the sample 
member responded to the second telephone survey, and the 10 interactions between 
the second telephone survey response indicator and the 10 school indicators. 
Additional potential predictors included basehne characteristics and outcomes 
measured in the in-person, first and second telephone surveys. ®® 



(rontinued) 

surveys, supplemented in a few cases by information from high school transcripts. For some sample members, 
we could not determine whether they had earned a diploma or G E D . For example, if a sample member did 
not respond to the third telephone survey after previously reporting having dropped out of school, we cannot 
be certain that the sample member did not earn a G E D or return to school and earn a diploma after their last 
response. For the two main outcomes measuring high school completion (as opposed to the alternative 
measures considered in the sensitivity analyses of Appendix F), we classify such a person as a nonrespondent. 
If, instead, a sample member did not respond to the third telephone survey after previously reporting having 
earned a diploma, we classify that person as a graduate and respondent. Alternatively, if a sample member did 
not respond to the third telephone survey after previously reporting having earned a G E D , we classify that 
person as a respondent who did not graduate from high school but did earn a G E D (see Appendix F). The 
sample sizes are thus the same for both high school completion measures and so we can use the same set of 
weights for both. However, because some third telephone survey nonrespondents' high school completion 
status is considered known (and thus not missing), a different set of weights must be used for the high school 
outcomes and all other outcomes. (We also derived additional sets of weights for two of the alternative 
measures considered in Appendix F, using the same methods as described here.) 

98 Our model selection procedure first estimated coefficients for the predictors forced into the model. 
Then, the procedure determined which excluded predictor had the largest adjusted chi-squared statistic for 
inclusion in the model. If the statistic was significant at the 75 percent confidence level, the procedure added 
the predictor to the model. The procedure never removed a predictor from the model. The procedure 
continued evaluating and adding excluded predictors until there was no excluded predictor that satisfied the 
criterion for inclusion. We examined alternative model selection procedures and determined that this 
procedure led to the best model. 

99 The baseline characteristics include an indicator for being male, an indicator for being black, an 
indicator for being Hispanic, two indicators for age when entering ninth grade (one for under 14 and one for 
over 14), two indicators for rank based on eighth-grade GPA (one for middle third and one for top third), 
percentile rank based on eighth-grade GPA, and the percentile rank squared. Other potential predictors 
included the interactions between any two baseline characteristics (except for the interaction between the two 
predictors based on eighth-grade G PA), and the interactions between any baseline characteristic and any school 
indicator. An exception to this was that the indicators for race and Hispanic origin were interacted with the 
school indicators for only the two schools with substantial diversity by race or Hispanic origin (Paschal in Fort 
Worth and D avis in Y akima). The outcomes measured in the in-person survey include an indicator for binge 
drinking in the 30 days before the survey, an indicator for using any illegal drug in the 30 days before the 
survey, an indicator for committing a crime in the 12 months before the survey, and an indicator for having 
ever been arrested or charged. The outcomes measured in the first and second telephone surveys include 
indicators for attending college at the time of each survey; indicators for attending postsecondaiy training at the 
time of each survey; indicators for attending having a job with health insurance at the time of each survey; 

(cnntinued) 
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2. We derived predicted response propensity scores based on sample 

members' cbaiacteristics using the best logit model for a sample 
member's evaluation group (QOP or control). 

3. We assigned a weight to a respondent egual to the inverse of the 

respondent's propensity score. We assigned a weight equal to zero to each 
nonrespondent. T o reduce the variabihty in weights and the resulting increase 
in variance of impact estimates, we trimmed weights to 3, that is, any weight 
greater than 3 was set equal to 3.^“ Weighting respondents based on inverse 
propensity scores ensured that we assigned a relatively large weight to a 
respondent who had a relatively low response propensity and, thus, resembled 
a nonrespondent. 

4. We ratio adjusted weights to sum to the number of respondents in each of 

the 22 weighting classes defined by the cross-classification of school (11 
categories) and experimental group (2 categories— QOP and control). 

Having developed these weights, we estimated an impact as the difference between the 
waited QOP group mean and the weighted control group mean. The details of the estimation 
of impacts and the variances of those impacts are presented in the next section.^“ 



(cnntinued) 

indicators for having a full-time job at the time of each survey; indicators for being in postsecondaiy training or 
in a job with health insurance at the time of each survey; indicators for attending or being accepted into college 
at the time of each survey; indicators for having one or more children at the time of each survey; indicators for 
binge drinking in the 30 days before each survey; indicators for frequent binge drinking in the 30 days before 
each survey; indicator for using illegal drugs in the 30 days before each survey; indicators for having committed 
a Clime in the 30 days before each survey; indicators for having been arrested or charged in the 30 days before 
each survey; and indicators for being on welfare at the time of each survey. Additional outcomes measured in 
just the first telephone survey are an indicator for stUl being in high school at the time of the survey; an 
indicator for having a high school diploma or GED or stiU being in high school at the time of the survey; an 
indicator for having been suspended from school; and an indicator for speaking English at home. EinaUy, 
additional outcomes measured only in the second telephone survey include an indicator for having a high 
school diploma; an indicator for having a high school diploma or G E D ; and an indicator for having completed 
at least two years of college. 

1“ This resulted in the trimming of 9 of the weights for the outcomes measuring high school completion, 
and 22 of the weights for aU of the other outcomes. 

101 In Appendix E, we assess the sensitivity of our results to whether and how we adjust for nonresponse. 
We find that our results are generally not sensitive. 
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Estimating Impacts and Variances of Impacts 



Impacts for Schools 

We estimated the impact for a school according to: 

impact = X —X 

school Q, school C, school 



where X is the outcome of interest Q and C denote the Q 0 P and control groups, and, for 
example: 



Q, school 



I 

i€.Q, school 




ieQ, school 



Wj is the weight for sample member i/°^ In other words, we estimated the impact for a 
school by subtracting the mean outcome among members of the control group for that 
school from the mean outcome among QOP enrollees for that school. Each sample 
member remained a member of the group to which he or she was originally assigned, 
regardless of subseguent behavior, such as transferring to another school, dropping out of 
school, or (for enrollees) dropping out of QOP. 

Treating the QOP group and the control group as independent samples from a 
superpopulation, we estimated the variance (the standard error sguared) of the school-level 
impact according to:^°^ 

var ( impact ) = var V -X = var V ] + var X 

\ school J \ Q.school C, school J \ Q, school J \ C, school J 



102 Because all of the sample members from a school had the same probability of assignment to the Q 0 P 
group, the only purpose of weighting is to adjust for unit nonresponse. We described earlier in this appendix 
how we derived weights. 

102 The basic idea is that we are not really interested in just the small population of youth who were 
eligible for random assignment. Rather, we would like to generalize to a "superpopulation" that includes other 
youth, including those who met the four QOP eligibility criteria (but were not selected for the initial sample) 
and those who would have been eligible in prior or subsequent academic years. If the group of youth eligible 
for random assignment were our population of interest, the Q 0 P and control means would be correlated 
(because the control group is the complement of the QOP group). However, that correlation is not 
estimable— without some simplifying assumption— because we observe each sample member in only one 
experimental state, that is, as either a Q 0 P enroUee or a control. 0 ne simplifying assumption is that the 
impact of QOP is additive and fixed (the same for aU youth). This assumption and the superpopulation 
approach lead to the same statistical procedure. 
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We estimated the variance of the Q 0 P group mean for a given school according to:^°^ 



var 



\ Q, 



school 



1 




( sr . 





w 

school j 



n 

Q,school 

n -1 

Q, school 



y w^X -X 

i \ i Q,school 

ieQ, school 



where ng^schooi is the number of responding sample members in the school's QOP group. 
When calculating any of the sums needed for estimating a mean or a variance, we includ^ 
only those sample members with valid (nonmissing) data for the outcome under 
consideration.^®^ 



Impacts for Sites 

We estimated the impact for a site according to: 

impact = V W impact 

site school school 

schools H 

site 

where Hate is the set of schools from which youth were selected for the QOP program 
operated at the site in question.^® In other words, we estimated the impact for a site by 
t^ng a weighted average of the impacts for the schools at that site. We based the school- 
level weights (the W ) on the allocation of slots observed in the demonstration. In fact, W ahod 
was the fraction of the site's QOP slots allocated to the particular school. Thus, Wahooi was 
1.00 for CoHinwood (Cleveland), Paschal (Fort Worth), Frankhn (Philadelphia), and Davis 
(Yakima); 0.50 for Anacostia (Washington, DC), Eastern (Washington, DC), Austin 
(Houston), and Yates (Houston); 0.35 for Carver (Memphis); 0.27 for Hamilton (Memphis); 
and 0.38 for HUlcrest (Memphis). This was our best estimate of how slots would have been 
allocated had the sites been part of an ongoing, national program. In such a program, as in 
the demonstration, CBOs in some sites would work with just one school, while CBOs in 
other sites would have the same number of slots, but work with two or three schools. In the 
latter case, the CBOs would likely allocate slots in the same way that the CBOs in the 
demonstration did. Note that for each site: 



104 A similar expression pertains for the variance of the control group mean. 

105 For all outcomes except those measuring high school completion, sample members who did not 
respond at aU to the third telephone survey were excluded because their weights were equal to zero. Sample 
members who responded to the survey but did not answer the question or questions relevant to the outcome 
were excluded from only those calculations for which they were missing data. The former group was 
substantially larger than the latter group for all the outcomes that we considered. We sought to compensate for 
the loss of the former group by weighting respondents, as described previously. For the outcomes measuring 
high school completion, there was no item nonresponse. 

106 Hgte consists of one school for Cleveland, Fort Worth, Philadelphia, and Yakima; two schools for 
Washington, D C, and Houston; and three schools for Memphis. 



E-7 




y w =i 

school 

schools H „ 
site 

This approach to weighting schools when calculating an impact estimate for a site implied, 
for example, that: 

impact =0.35ximpact + 0.21 ximpact + 0.3Sximpact 

Memphis Carver Hamilton Hillcrest 

Treating the allocation of QOP slots across schools within a site as fixed, we estimated 
the variance of the site- level impact according to: 

war {impact )= V vac [impact ) 

\ site / school V school / 

schools H 

site 

This expression reflects the fact that for each site, we had the full population of schools 
from which youth were selected and the fact that random assignment was carried out 
independently in each school. 

Impacts for the Whole Demonstration 

We estimated the impact for the whole QOP demonstration according to: 

7 

impact =} A impact 

demo site site 

site = 1 



where 



7 




site — 1 



We assumed that = 1/7 for all sites. Thus, we estimated the impact for the whole 
demonstration by taking the simple average of the seven site impacts. Our egual weighting 
of sites was based on the belief— or best guess— that if Q 0 P were implemented as an 
ongoing, national program, CBOs would have roughly egual numbers of QOP slots. The 
relatively small sizes of the Washington, DC, and Ford-funded programs in the 
demonstration were due to circumstances that we do not think would be rephcated in a 
regular program.^®’ 



10’ The Ford Foundation wanted to fund two sites, but at only half of the size of DOL-funded sites, an 
outcome that would be unlikely to occur in a program that is fuUy funded by the federal government. The 
Washington, DC, site was allocated 100 QOP slots, but given the short duration of the demonstration and the 
one-year delay in beginning program operations in the site, efforts to identify eligible youth were halted at a 
third QOP school that would have had 20 slots. This decision was not made early enough to increase the 
number of slots at the two remaining QOP schools. 
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For deriving all of the estimates presented in this report, we assumed that the collection 
of sites in the Q 0 P demonstration was a fixed set, that is, a population. Thus, we estimated 
the variance of the demonstration-level impact according to: 

7 

wax: [impact = V war [impact 

V demo / site V site / 

site — 1 

Although the sites in the demonstration were not really a population, they were also not 
a probability sample. Nevertheless, if statistically significant impact estimates from the 
demonstration are to be useful for informing policy, the demonstration sites must be 
approximately representative of a population of potential sites. Then, we would want to 
treat the demonstration sites as a random sample (of size seven), and estimate the "total" 
variance of an impact estimate. The total variance has both a within-site component and a 
between-site component. The within-site component reflects the sampling error from 
selecting samples of youth in each site, and is captured by the expression already given for 
the variance of the demonstration-level impact. The between-site component reflects the 
differences among the impacts for the different sites. Although we might have preferred to 
obtain estimates of total variances, we cannot estimate total variances very precisely because 
there were only seven sites in the demonstration. In fact, we discovered in the analysis of 
short-term impacts that for a large majority of impacts, the estimated total variance was 
smaller— often substantially smaller— than the estimated within-site component of variance. 
Because we prefer a well- estimated within-site component of variance to a poorly estimated 
total variance, we present the former as our variance estimates. 

We conducted t- tests to determine whether estimated impacts were significantly 
different from zero. For a t-test, we calculated a t-statistic by dividing an impact estimate by 
its standard error. The standard error is the sguare root of the variance, and the variance 
was estimated according to the relevant expression given in this appendix. 
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Appendix F 
Sensitivity Analyses 
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In this appendix, we assess the sensitivity of the impact estimates to alternative 
estimation approaches. In particular, we assess the sensitivity to: 

• A Iteraative approaches to measuring hi^ school completion. T o gauge the potential 
importance of concerns about both nonresponse and the accuracy of responses, 
we have assessed the sensitivity of the impact estimates to alternative ways of 
measuring high school completion, some of which infer graduation and GED 
completion status based on certain assumptions that we have made when the 
available data are incomplete or seemingly inconsistent. 

• A Itemative modds of nonresponse. T o assess the sensitivity of our results to whether 
and how we adjust for nonresponse, we estimated impacts using alternative 
methods to adjust for nonresponse. 

• U sing regression methods to adjust the impad estimates for random differences between the 
QOP pup and the control pup. Although the difference-of-means estimates 
presented in this report are unbiased, they may have been affected by purely 
random differences between the baseline characteristics of Q 0 P enrollees and 
the baseline characteristics of members of the control group. Therefore, we 
adjusted for such differences using regression methods. 

In each case, we determined whether our conclusions would have been different had 
they been based on estimates derived using the alternative approaches. We found that our 
conclusions are generally robust. 

In addition to these sensitivity analyses, we assessed whether our impact estimates for 
the whole QOP demonstration are sensitive to the inclusion or exclusion of the Washington, 
D C site. We undertook this assessment for two reasons. First, the difference between the 
treatment and control group response rates in the Washington, DC site was relatively large 
(12 percentage points, as documented in Appendix C). Second, program operations began a 
year later and sample members are typically a year younger in the Washington, D C site than 
in the other six sites. As in the other sensitivity analyses, this sensitivity analysis revealed that 
our conclusions are generally robust. 



The Sensitivity of Impact Estimatesto Alternative approachesto 
Defining High School Graduation and GED completion Status 



We determined a sample member's high school graduation and GED completion status 
based primarily on that person's responses to guestions asked during the first, second, and 
third telephone surveys, supplemented in a few cases by information from high school 
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transcripts/®^ Such an approach to measuring these outcomes— the same basic approach 
taken with the National Education Longitudinal Study— raises two concerns. First, as noted 
in Appendix C, some sample members did not respond to the surveys or had incomplete 
transcript data. Second, some sample members might not have provided accurate responses 
to the guestions about high school completion. 

To reduce the effects of survey nonresponse, we developed weights that adjust for 
differences between respondents and nonrespondents.^® Furthermore, to gauge the 
potential importance of concerns about both nonresponse and the accuracy of responses, we 
have also assessed the sensitivity of the impact estimates to alternative ways of measuring 
high school completion, some of which infer graduation and GED completion status based 
on certain assumptions that we have made when the available data are incomplete or 
seemingly inconsistent. 

The high school and GED completion measures presented in the main text of the 
report use information on third telephone survey respondents obtained in that survey as well 
as some nonrespondents who had reported earning a high school diploma or GED in a 
previous survey. In particular, individuals who had reported earning a diploma are classified 
as graduates. Individuals who had reported earning a GED are classified as not having 
earned a high school diploma but as having earned a G E D .“® 

Our first sensitivity analysis of high school completion (the "First Alternative" in Tables 
F.l to F.3) uses just the sample members who responded to the third telephone survey, the 
same sample on which all odier outcomes reported in the main text are based. AH sample 
members' responses to the third telephone survey regarding their high school completion 
status are assumed to be accurate. 

The second and third sensitivity analysis measures incorporate additional information 
from previous surveys and do not necessarily treat all third telephone survey responses as 
fully accurate. There are two types of individuals whose measured outcomes are affected for 
the second sensitivity analysis (resulting impacts shown as the "Second Alternative" in 
Tables F.l to F.3). The first are sample members whose third telephone survey responses 
are inconsistent with previous responses. These individuals were classified according to a 
case-by-case examination of all of the available data. The second set of individuals whose 
classifications were modified are sample members whose available transcript data 
contradicted their report of having earned a high school diploma or strongly suggested that 



108 por 10 sample members who had not responded to the telephone surveys, the sample members' 
transcripts indicated that they had graduated from high school. For most sample members who did not 
respond to our surveys, transcript data were incomplete. 

109 We describe our method for deriving weights in Appendix E. Later in this appendix, we assess the 
sensitivity of estimates to alternative approaches for obtaining weights. 

110 This may underestimate high school graduation rates if some individuals earned a high school diploma 
after earning a GED. We believe that behavior is sufficiently rare to justify the assumption that individuals 
who had earned a G ED at one point in time did not later earn a high school diploma. 
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the person had not graduated. These individuals are classified as nongraduates. Their G ED 
status is assumed to be missing since some may have obtained a G ED after their last survey 
response. 

The final sensitivity measure (the “Third Alternative" in Tables E.l to E.3) is the same 
as the second alternative, and does not further modify the measure of whether sample 
members received a G ED , but classifies individuals who were very unlikely to have earned a 
high school diploma as nongraduates.^“ This measure assumes that individuals who had not 
earned a high school diploma within about three years of scheduled high school graduation, 
around age 21, are unhkely to go back and receive a high school diploma. However, 
obtaining a GED after that point in time is plausible and so the measure of GED 
completion for these sample members is treated as missing. 

According to Table E.l, the estimated impacts of QOP on high school graduation and 
GED completion are not sensitive to how we measure graduation and GED completion. 
All of the estimates imply that QOP did not significantly increase the likehhood of 
graduating from high school or the likelihood of graduating from high school or earning a 
G ED Similarly, the site-specific impact estimates are also generally not sensitive to how 
we measure high school completion (see T able E.2). Cleveland is the only site to show a 
significant impact on high school completion, and although the impact on completion of a 
high school diploma is somewhat sensitive to how that completion is measured, the impact 
on completion of a diploma or G E D is not sensitive. T able E.3 indicates that the impact on 
high school completion for younger enroHees is moderately sensitive to how we measure 
that completion, with some of the sensitivity analysis measures leading to larger and more 
significant impacts than in the main analysis but one measure yielding a non- significant 
impact. 

The alternative estimates in Table E.l imply the same conclusion and are consistent 
with results reported in Schirm and Rodriguez- Planas (2004). However, these results do 
differ from those in Maxfield et al. 2003b, which indicated that as of the first telephone 
survey, QOP significantly increased by seven percentage points the likehhood of earning a 
diploma. One possible explanation for the different findings is that QOP enrollees were 
more likely than control group members to graduate on time— in four years— or in one extra 
semester but the control group members subsequently caught up by remaining in school 
longer and graduating in, say, five full years. As noted in Schirm and Rodriguez-Planas 
(2004), however, estimates based on ah of the data cohected up to about three years after 
sample members' scheduled high school graduation (the time of the second telephone 



111 This group consists of two types of sampie members. The first are individuais who iast responded to 
the first teiephone survey and at that time had been out of schooi for one entire academic year and part of 
another, had not progressed beyond tenth grade, or were not on a pace to graduate from high schooi in five 
years based on cr^its received. The second type are individuais who did not respond to the third teiephone 
survey and had reported not having graduated from high schooi in the second teiephone survey. 

112 Given that the sampie members entered— and often remained in— high schoois with high dropout 
rates (over 40 percent) the means in Tabie F.l seem piausibie. 
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survey), reveal that QOP did not significantly increase the hkelihood of graduating on time 
or in less than five years/^^ 



Table F.1. Impacts on High School Completion Using Alternative Approaches to Measuring 
Graduation and GED Completion 



Qutcome 


QQP Group Mean 
(percentage) 


Contrel Group 
Mean 

(percentage) 


Impact 

(percentage points) 


Primary Measures 


Received HS diploma 


60 


60 


0 


Received HS diploma or GED 


78 


75 


2 


First Alternative Measures 


Received HS diploma 


58 


57 


1 


Received HS diploma or GED 


73 


71 


2 


Second Alternative Measures 


Received HS diploma 


54 


53 


1 


Received HS diploma or GED 


74 


70 


4 


Third Alternative Measures 


Received HS diploma 


52 


48 


4 



Source: Telephone surveys and transcripts. 

Note: Each impact was derived by subtracting the control group mean from the QOP group mean prior 

to rounding those means; thus, an impact might not equal the difference between the rounded 
means that are displayed. Estimates were obtained using weights to adjust for differences 
between respondents and nonrespondents in baseline characteristics, response rates to the first 
and second telephone surveys, and outcomes measured in the in-person, first, and second 
telephone surveys. The evaluation sample had 580 QOP enrollees and 489 controls. 

Note The third alternative measure does not change any sample members’ classification of having 
received a high school diploma or GED, and thus only the high school diploma measure is 
reported. 

* Estimate significantly different from zero at the 90% confidence level, two-tailed test 

** Estimate significantly different from zero at the 95% confidence level, two-tailed test 

*** Estimate significantly different from zero at the 99% confidence level, two-tailed test 



113 See Schirm and Rodriguez- Planas (2004) for a full comparison of the results in that report and those in 
Maxfield et al. (2003b). 
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Table F.2. Impacts on High School Completion Using Aiternative Approaches to Measuring Graduation and GED Completion by Site (Percentage Points) 
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Table F.3. Impacts on High School Completion Using Alternative Approaches to Measuring 
Graduation and GED Completion, Students 14 or Younger When Entering Ninth Grade 



Outcome 


QOP Group Mean 
(percentage) 


Control Group 
Mean 

(percentage) 


Impact 

(percentage points) 


Primary Measures 


Received HS diploma 


70 


63 


7* 


Received HS diploma or GED 


87 


81 


6* 


First Alternative Measures 


Received HS diploma 


68 


61 


7 


Received HS diploma or GED 


83 


78 


6 


Second Alternative Measures 


Received HS diploma 


66 


56 


9** 


Received HS diploma or GED 


85 


77 


8** 


Third Alternative Measures 


Received HS diploma 


63 


51 


■j -j *** 



Source: Telephone surveys and transcripts. 

Note: Each impact was derived by subtracting the control group mean from the QOP group mean prior 

to rounding those means; thus, an impact might not equal the difference between the rounded 
means that are displayed. Estimates were obtained using weights to adjust for differences 
between respondents and nonrespondents in baseline characteristics, response rates to the first 
and second telephone surveys, and outcomes measured in the in-person, first, and second 
telephone surveys. The evaluation sample had 580 QOP enrollees and 489 controls. 

Note The third alternative measure does not change any sample members’ classification of having 
received a high school diploma or GED, and thus only the high school diploma measure is 
reported. 

* Estimate significantly different from zero at the 90% confidence level, two-tailed test 

** Estimate significantly different from zero at the 95% confidence level, two-tailed test 

*** Estimate significantly different from zero at the 99% confidence level, two-tailed test 
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The Effects of Nonresponse and the Sensitivity of Impact Estimatesto 
THE Approach for Adjusting for Nonresponse 



To assess the sensitivity of our results to whether and how we adjust for nonresponse, 
we estimated impacts using alternative weights to adjust for nonresponse."^ Tables F.4 
through F.7 present the impact estimates that we obtained using two alternative methods of 
assessing the effects of nonresponse and, particularly, of differential nonresponse between 
the QOP and control groups. Our preferred estimates, which are those presented in the 
tables in the main part of the report, are also presented in Tables F.4 through F.7. Our 
preferred estimates were derived using weights that adjust for differences between 
respondents and nonrespondents in baseline characteristics, response rates to the first and 
second telephone surveys, and outcomes measured in the in-person, first, and second 
telephone surveys. The construction of those weights is described in Appendix E . 

The first alternative results presented are unweighted estimates, presented for 
comparison purposes. The second dtemative results can be examined to assess the effects 
of differentia nonresponse between the QOP and control groups. They were derived by 
making the response rate for the Q 0 P group egual to the response rate for the control 
group within each of the 1 1 Q 0 P schools. That is, if the Q 0 P group had a higher response 
rate, we treated enough QOP group respondents as nonrespondents to lower the implied 
response rate to the level of the control group. The QOP group respondents that were 
treated as nonrespondents were the last ones to respond to the survey. 

Comparing the alternative impact estimates suggests that whether and how we weight to 
adjust for nonresponse might affect only two of our conclusions. The first is that Q 0 P did 
not have impacts on engagement in any postsecondaiy education or training or on 
completion of two years of postsecondaiy education or training. The two alternative 
estimates imply impacts of 7 percentage points on both of these outcomes. The second is 
that QOP increased the likelihood that enrollees had committed a crime in the past three 
months. The two alternative estimates are not statistically significant. However, the 
significant impact on the likelihood of having been arrested or charged in the past two years 
is statistically significant using all three approaches."^ 



Appendix E describes the methodology we followed to develop person-level weights to adjust for the 
potential effects of unit nonresponse. 

115 When significance levels are adjusted for the multiple comparisons that are being performed 
simultaneously, the significant impacts on postsecondaiy attainment seen in the unweighted analyses are no 
longer statistically significant. Likewise, when such adjustments are done for the outcomes measuring criminal 
activity, the weighted impact on having committed a crime in the past three months is no longer statistically 
significant, although the impact on having been arrested or charged in the past two yeans remains significant. 
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Table F.4. Impacts on High School Completion Obtained With Alternative Approaches to 
Adjusting for Nonresponse (Percentage Points) 





Unweighted 


Unweighted 


Weighted 


Outcome 


Estimate 


Estimate® 


estimates 


Received HS diploma 


2 


2 


0 


Received HS diploma or GED 


3 


2 


2 



Source: Telephone surveys and transcripts. 

Note: Each impact was derived by subtracting the control group mean from the OOP group mean. The 

weights used to derive the weighted estimates adjust for differences between respondents and 
nonrespondents in baseline characteristics, response rates to the first and second telephone 
surveys, and outcomes measured in the in-person, first, and second telephone surveys. The 
evaluation sample had 580 OOP enrollees and 489 controls. 

® These estimates were derived by making the response rate for the QOP group equal to the response rate 
for the control group for each of the 1 1 schools. That is, if the QOP group had a higher response 
rate, we treated enough QOP group respondents as nonrespondents to lower the implied 
response rate to the level of the control group. The QOP group respondents that were treated as 
nonrespondents were the last ones to respond to the survey. 

* Estimate significantly different from zero at the 90% confidence level, two-tailed test 

** Estimate significantly different from zero at the 95% confidence level, two-tailed test 

*** Estimate significantly different from zero at the 99% confidence level, two-tailed test 
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Table F.5. Impacts on Postsecondary Attainment Obtained With Alternative Approaches to Adjusting for Nonresponse (Percentage Points) 
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Table F.6. Impacts On Employment and Earnings Obtained With Alternative Approaches To Adjusting For Nonresponse (Percentage Points) 

Unweighted Unweighted Weighted 

Outcome Estimates Estimates® Estimates 

Currently employed 1 -0 -1 

Currently unemployed -2 -2 -1 
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Adjusting For Nonresponse (Percentage Points) 
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The Sensitivity of Impact Estimates to Regression Adjustment 



Baseline Differences Between the QOP and Control Groups 

According to T able F.8, there was just one statistically significant difference between the 
means of baseline characteristics for the QOP and control groups for the whole 
demonstration. Compared with the control group, the QOP group had fewer youth in the 
middle third of the eighth-grade GPA distribution. When we examined QOP and control 
group means for schools— the level at which we conducted random assignment— we found 
only a few significant differences (not shown in T able F.8). For example, compared with the 
school's control group, the QOP group from Austin High School (Houston) had more 
youth over age 14 and fewer youth from the top third of the grade distribution; the QOP 
group from Y ates High School (Houston) had fewer youth from the bottom third and more 
youth from the top third of the grade distribution; and the QOP group from HUlcrest High 
School (Memphis) had more youth from the middle third of the grade distribution. 

Deriving Regression-Adjusted Impact Estimates 

Our regression model included 37 variables: 11 school indicators, 11 interactions 
between a QO P/ control indicator and the 11 school indicators, an indicator for being male, 
an indicator for being over age 14 when entering ninth grade, an indicator for being in the 
middle third of the eighth-grade G PA distribution, an indicator for being in the top third of 
the eight-grade G PA distribution, five variables obtained by interacting the last four basehne 
characteristic variables, and six additional variables obtained by interacting some of the 
basehne characteristic variables with some of the school indicators."® We estimated the 
parameters of this regression model for each outcome considered. For binaiy outcomes we 
used logit regression models and for continuous outcomes we used linear regression models. 
For the continuous outcomes we obtained impact and variance estimates for schools from 
the estimated distribution of the coefficients on the QOP/ control indicator and school 
interaction terms. For the binaiy outcomes we obtained impact and variance estimates for 
schools from probabilities predicted by the logit model for the outcome under 
consideration."’ After deriving school-level estimates, we derived site- and demonstration- 
level estimates using the expressions in Appendix E . 



116 The 11 school indicators and interactions between a QOP/ control indicator and the 11 school 
indicators were included to allow the estimation of school-level impacts (see Appendix E). AU baseline 
characteristics available, as well as interactions between those characteristics, were dso included. The last six 
variables were included to adjust for significant differences between the Q 0 P and control groups in some of 
the QOP schools. 

11’ Suppose that we are estimating a regression-adjusted impact on high school graduation. Then, for 
every sample member from a given school, we used the estimated logit model for high school graduation to 
obtain four predicted probabilities while ignoring the sample member's actual QOP/ control status: (1) the 
probability of graduation if the sample member is a control, (2) the probability of graduation if the sample 
member is a Q 0 P enrollee and the effect of Q 0 P is measured by the coefficient on the interaction between 

(rontinued) 
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Regression- Adjusted Impact Estimates 

Tables F.9 through F.12 present difference-of-means and regression- adjusted impact 
estimates. Comparing the alternative impact estimates suggests that regression adjustment 
affects only three of our conclusions— that QOP had no impact on enrollment in 
postsecondaiy education or training, and that it increased the hkelihoods of committing a 
crime in the three months before the survey and of being arrested or charged in the previous 
two years. Table F.IO shows that regression adjustment increased by one percentage 
point— from an insignificant six to a significant seven percentage points— the impact on 
ever being engaged in postsecondaiy education or training of any type. A similar change is 
seen on the measure of postsecondaiy engagement that includes enrollment in Job Corps.^^*^ 
Table F.12 shows that the regression-adjusted impacts on the hkelihoods of having 
committed a crime in the past three months or of having been arrested or charged in the 
past two years are not statistically significant in contrast to the difference-of-means 
estimates, which showed significant impacts on these outcomes. All other impacts that were 
insignificant according to difference-of-means estimates are also insignificant according to 
regression-adjusted estimates. 



(continued) 

the indicator for the sample member's school and the Q 0 P/ control indicator, (3) the probability of graduation 
if the sample member is a QOP enroUee and the effect of QOP is measured by the coefficient on the 
interaction between the indicator for the sample member's school and the Q 0 P/ control indicator plus the 
standard error of the coefficient, and (4) the probability of graduation if the sample member is a Q 0 P enroUee 
and the effect of Q 0 P is measured by the coefficient on the interaction between the indicator for the sample 
member's school and the QOP/ control indicator minus the standard error of the coefficient. N ext, we 
calculated the mean for each of these probabilities across all of the sample member from the school. The 
second mean minus the first mean was our impact estimate for the school We estimated the variance of the 
impact by squaring the difference between the tiitrd and fourth means and dividing by four. 

118 When significance levels are adjusted for the multiple comparisons that are being performed 
simultaneously, the significant impacts on postsecondaiy attainment seen in the regression adjusted analyses are 
no longer statistically significant. 
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Table F.8. Group Means for Baseline Characteristics (Percentages) 







Means 




Baseline Characteristic 


OOP Group 




Control Group 


Male 


52 




56 


Age when entering ninth grade 


< 14 


11 




11 


14 


53 




57 


> 14 


36 




31 


Hispanic 


26 




26 


Black 


68 




68 


Rank based on eighth-grade GPA 


Bottom Third 


37 




34 


Middle Third 


3F 




36^ 


Top Third 


32 




30 



Source: Baseline database. 

Note: The evaluation sample had 580 OOP enrollees and 489 controls. 

t Significantly different from the mean for the other group at the 90% confidence level, two-tailed test 

ft Significantly different from the mean for the other group at the 95% confidence level, two-tailed test 

ttt Significantly different from the mean for the other group at the 99% confidence level, two-tailed test 
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Table F.9. 



Difference-of-Means Versus Regression-Adjusted Impacts on High School 
Completion (Percentage Points) 



Outcome 


Difference of Means 


Regression Adjusted 


Received HS diploma 


0 


3 


Received HS diploma or GED 


2 


4 



Source: Telephone surveys and transcripts. 

Note: Estimates were obtained using weights to adjust for differences between respondents and 

nonrespondents in baseline characteristics, response rates to the first and second telephone 
surveys, and outcomes measured in the in-person, first, and second telephone surveys. The 
evaluation sample had 580 QOP enrollees and 489 controls. 

* Estimate significantly different from zero at the 90% confidence level, two-tailed test 

** Estimate significantly different from zero at the 95% confidence level, two-tailed test 

*** Estimate significantly different from zero at the 99% confidence level, two-tailed test 
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Table F. 10 . Difference-of-Means Versus Regression-Adjusted Impacts on Postsecondary Attainment (Percentage Points) 
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Table F.11. Difference-of-Means Versus Regression-Adjusted Impacts on Employment and Earnings (Percentage Points, Unless Stated 
Otherwise) 
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Table F.12. Difference-of-Means Versus Regression- Adjusted Impacts on Risky Behaviors, 
Physical and Mental Well-Being, and Family Life (Percentage Points) 



Outcome 


Difference of Means 


Regression Adjusted 


Smoked cigarettes or used tobacco in past month 
Smoked cigarettes or used tobacco daily in past 


0 


-0 


month 


-2 


-2 


Binge drinking in past month 


0 


0 


Binge drinking on 8 or more days in past month 


3 


3 


Used an illegal drug in past month 


-0 


-1 


Committed a crime in past 3 months 


3* 


2 


Committed a crime in past 2 years 


5 


4 


Arrested or charged in past 2 years 


6** 


6 


Convicted or pled guilty in past 2 years 

Served time in jail, prison, or detention home in past 2 


2 


2 


years 


1 


1 


Self-reported health is fair, poor, or very poor 
Physical or mental condition limited activities quite a 


2 


1 


lot or could not work because of these limitations 


1 


0 


Had first child before age 18 


2 


2 


Currently living with natural children, but no spouse 


1 


0 


Have children with whom not currently living 
Have child with whom not living and not providing any 


1 


1 


regular child support 


-2 


-2 


Currently receiving welfare 


1 


1 


Currently receiving food stamps 


2 


1 


Currently receiving welfare or food stamps 


3 


2 



Source: Telephone survey. 



Note: Estimates were obtained using weights to adjust for differences between respondents and 

nonrespondents in baseline characteristics, response rates to the first and second telephone 
surveys, and outcomes measured in the in-person, first, and second telephone surveys. The 
evaluation sample had 580 QOP enrollees and 489 controls. 

* Estimate significantly different from zero at the 90% confidence level, two-tailed test 

** Estimate significantly different from zero at the 95% confidence level, two-tailed test 

*** Estimate significantly different from zero at the 99% confidence level, two-tailed test 
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The Sensitivity of Impact Estimates to Including and Excluding the 

Washington, DC Site 



Tables F.13 through F.16 present two sets of impact estimates: (1) the estimates 
obtained when the Washington, D C site is included and (2) the estimates obtained when that 
site is excluded. The former are the estimates presented in the main text of this report. 

As noted above, one reason for undertaking this sensitivity analysis is that program 
operations began a year later in the Washington, DC site than in the other sites and sample 
members are typically a year younger in the Washington, D C site. This does not threaten the 
internal validity of the impact estimates for the Washington, DC site. However, if all else 
were egual, the estimates for that site might still differ from the estimates for the other sites 
if the impacts for some outcomes tend to rise or fall as time passes or as sample members 
age.“® Among the outcomes that might be susceptible to impacts changing over time are the 
high school completion outcome that counts the receipt of a G ED as a form of completion 
and the outcomes pertaining to postsecondaiy education and training if some young adults 
who did not receive a diploma take several years to earn a GED while others who have 
completed high school delay enrollment in a postsecondaiy education or training program or 
enroll part time and take several years to complete the program. For the impacts pertaining 
to risky behaviors and family life, impacts might change with age. 

Although these effects are conceivable, it is difficult to predict their direction, that is, 
whether a given impact will rise or fall with time or the age of sample members. According 
to Tables F.13 through F.16, we obtain for most outcomes essentMly the same results when 
the Washington, DC site is included as when it is excluded. With only a few exceptions, 
significant impacts remain significant, and insignificant impacts remain insignificant. When 
differences are found, they tend to be in the direction of showing larger detrimental impacts 
when the Washington, D C site is excluded. For example, excluding the Washington, D C site 
increases the impact on freguent binge drinking from an insignificant 3-percentage-point 
increase to a significant 5-percentage-point increase and also increases the detrimental 
impacts on criminal activity. D espite our previous conjecture, we find that excluding the 
Washington, D C site does not affect the impact on the hkelihood of high school completion 
via a diploma or GED or most of the impacts pertaining to postsecondaiy education or 
training. 



119 The third telephone survey was conducted from January 2005 to September 2005 in aU sites. Thus, it 
began three years after the end of the demonstration and nearly five years after scheduled graduation in the 
Washington, D C site. In the other sites, the survey began four years rfter the end of the demonstration and 
nearly six years after scheduled graduation. 
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Table F.13. Impacts on High School Completion When the Washington, DC Site is inciuded and 
When it is Excluded (Percentage Points) 





Washington, DC Site 


Outcome 


Included 


Excluded 


Received HS diploma 


0 


-1 


Received HS diploma or GED 


2 


1 



Source: Telephone surveys and transcripts. 

Note: Each impact was derived by subtracting the control group mean from the OOP group 

mean. Estimates were obtained using weights to adjust for differences between 
respondents and nonrespondents in baseline characteristics, response rates to the first 
and second telephone surveys, and outcomes measured in the in-person, first, and 
second telephone surveys. The evaluation sample had 580 QOP enrollees and 489 
controls. 

t Significantly different from the impact on all other sample members at the 90% confidence 

level, two-tailed test 

ft Significantly different from the impact on all other sample members at the 95% confidence 

level, two-tailed test 

ttt Significantly different from the impact on all other sample members at the 99% confidence 

level, two-tailed test 

* Estimate significantly different from zero at the 90% confidence level, two-tailed test 

** Estimate significantly different from zero at the 95% confidence level, two-tailed test 

*** Estimate significantly different from zero at the 99% confidence level, two-tailed test 
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Table F.14. Impacts on Postsecondary Attainment When the Washington, DC Site is Included and 
When it is Excluded (Percentage Points) 

Washington, DC Site 



Outcome Included Excluded 

Ever attended or currently attending a 4-year college 1 2 

Completed at least 1 year at a 4-year college 1 3 

Completed at least 2 years at a 4-year college 1 2 

Earned a bachelor’s degree 1 1 

Ever attended or currently attending a 2- or 4-year college 4 4 

Completed at least 1 year at a 2- or 4-year college 2 2 

Completed at least 2 years at a 2- or 4-year college 2 2 

Earned a bachelor’s or associate’s degree -1 -0 

Ever or currently in college, voc/tech school, an apprenticeship, or the military 6 4 

Completed 2 years of college or military service, completed voc/tech school or 

an apprenticeship, or honorably discharged from the military 5 3^ 

Completed an associate’s or bachelor’s degree, voc/tech school or an 
apprenticeship, in the military for more than 2 years, or honorably discharged 

from the military 2 1 

Ever or currently in college, voc/tech school, an apprenticeship, the military, or 

Job Corps 6 4 

Currently in a 4-year college -2 -2 

Currently in a 2- or 4-year college -4 -4 

Currently in college, voc/tech school, an apprenticeship, or the military -1 -1 



Source: Telephone survey. 

Note: Each impact was derived by subtracting the control group mean from the OOP group mean. 

Estimates were obtained using weights to adjust for differences between respondents and 
nonrespondents in baseline characteristics, response rates to the first and second telephone 
surveys, and outcomes measured in the in-person, first, and second telephone surveys. The 
evaluation sample had 580 OOP enrollees and 489 controls. 

t Significantly different from the impact on all other sample members at the 90% confidence level, 

two-tailed test 

ft Significantly different from the impact on all other sample members at the 95% confidence level, 

two-tailed test 

ttt Significantly different from the impact on all other sample members at the 99% confidence level, 
two-tailed test 

* Estimate significantly different from zero at the 90% confidence level, two-tailed test 

** Estimate significantly different from zero at the 95% confidence level, two-tailed test 

*** Estimate significantly different from zero at the 99% confidence level, two-tailed test 
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Table F.15. Impacts on Employment and Earnings When the Washington, DC Site is Included and 
When it is Excluded (Percentage Points, Unless Otherwise indicated) 



Outcome 


Washington, DC Site 
Included Excluded 


Currently employed 


-1 


0 


Currently unemployed 


-1 


-2 


Currently out of labor force 


2 


2 


Currently employed or in college, voc/tech school, an apprenticeship, or the 
military 


1 


0 


Ever employed 


0 


-0 


Employed in past 12 months 


-1 


-1 


Percentage of weeks employed in past 12 months (percentage of weeks) 


-2 


-1 


Number of jobs in past 1 2 months (number of jobs) 


0.0 


0.0 


Tenure at current job (months) 


-2 


-1 


Usual number of hours worked per week in all current jobs (hours) 


-0 


0 


Works at least 35 hours per week at main current job 


-0 


0 


Total earnings in past 12 months (dollars) 


-522 


-549 


Hourly earnings at main current job (dollars) 


-1.20 


-1.57 


Has a job with health insurance 


-3 


-1 


Has a job with paid time off 


-2 


-0 


Has a job with a pension or retirement benefits 


-1 


0 



Source: Telephone survey. 

Note: Each impact was derived by subtracting the control group mean from the OOP group mean. 

Estimates were obtained using weights to adjust for differences between respondents and 
nonrespondents in baseline characteristics, response rates to the first and second telephone 
surveys, and outcomes measured in the in-person, first, and second telephone surveys. The 
evaluation sample had 580 OOP enrollees and 489 controls. 

t Significantly different from the impact on all other sample members at the 90% confidence level, 

two-tailed test 

ft Significantly different from the impact on all other sample members at the 95% confidence level, 

two-tailed test 

ttt Significantly different from the impact on all other sample members at the 99% confidence level, 
two-tailed test 

* Estimate significantly different from zero at the 90% confidence level, two-tailed test 

** Estimate significantly different from zero at the 95% confidence level, two-tailed test 

*** Estimate significantly different from zero at the 99% confidence level, two-tailed test 



F-24 




Table F.16. Impacts on Risky Behaviors, Physical and Mental Well-Being, and Family Life When 
the Washington, DC Site is included and When it is Exciuded (Percentage Points) 



Outcome 


Washington, DC Site 
Included Excluded 


Smoked cigarettes or used tobacco in past month 


0 


-0 


Smoked cigarettes or used tobacco daily in past month 


-2 


-1 


Binge drinking in past month 


0 


1 


Binge drinking on 8 or more days in past month 


3 


5**tt 


Used an illegal drug in past month 


-0 


-1 


Committed a crime in past 3 months 


3* 


4**t 


Committed a crime in past 2 years 


5 


6* 


Arrested or charged in past 2 years 


6** 


y*** 


Convicted or pled guilty in past 2 years 


2 


2 


Served time in jail, prison, or detention home in past 2 years 


1 


1 


Self-reported health is fair, poor, or very poor 

Physical or mental condition limited activities quite a lot or could not work 


2 


1 


because of these limitations 


1 


1 


Had first child before age 18 


2 


2 


Currently living with natural children, but no spouse 


1 


-0 


Have children with whom not currently living 


1 


3^ 


Have child with whom not living and not providing any regular child support 


-2 


-1 


Currently receiving welfare 


1 


1 


Currently receiving food stamps 


2 


0 


Currently receiving welfare or food stamps 


3 


1 



Source: Telephone survey. 

Note: Each impact was derived by subtracting the control group mean from the QOP group mean. 

Estimates were obtained using weights to adjust for differences between respondents and 
nonrespondents in baseline characteristics, response rates to the first and second telephone 
surveys, and outcomes measured in the in-person and first two telephone surveys. The 
evaluation sample had 580 QOP enrollees and 489 controls. 

t Significantly different from the impact on all other sample members at the 90% confidence level, 

two-tailed test 

ft Significantly different from the impact on all other sample members at the 95% confidence level, 

two-tailed test 

ttt Significantly different from the impact on all other sample members at the 99% confidence level, 
two-tailed test 

* Estimate significantly different from zero at the 90% confidence level, two-tailed test 

** Estimate significantly different from zero at the 95% confidence level, two-tailed test 

*** Estimate significantly different from zero at the 99% confidence level, two-tailed test 
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QOP AND Control Group Meansfor Subgroups 
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Table G.1. QOP Group Means Pertaining to High School Completion by Sex (Percentages) 







Means 




Outcome 


Male 


Female 


Total Sample 


Received HS diploma 


56 


65 


60 


Received HS diploma or GED 


78 


77 


78 



Source: Telephone surveys and transcripts. 

Note: Means were estimated using weights to adjust for differences between respondents and 

nonrespondents in baseline characteristics, response rates to the first and secend telephene 
surveys, and outcomes measured in the in-person, first, and second telephone surveys. The 
evaluation sample had 580 QOP enrollees and 489 controls. 



Table G.2. Control Group Means Pertaining to High Schooi Compietion by Sex (Percentages) 







Means 




Outcome 


Male 


Female 


Total Sample 


Received HS diploma 


54 


66 


60 


Received HS diploma or GED 


71 


79 


75 



Source: Telephone surveys and transcripts. 

Note: Means were estimated using weights to adjust for differences between respondents and 

nonrespondents in baseline characteristics, response rates to the first and second telephone 
surveys, and outcomes measured in the in-person, first, and second telephone surveys. The 
evaluation sample had 580 QOP enrollees and 489 controls. 
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Table G.3. QOP Group Means Pertaining to Postsecondary Attainment by Sex (Percentages) 
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Table G.4. Control Group Means Pertaining to Postsecondary Attainment by Sex (Percentages) 
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QOP enrollees and 489 controls. 




Table G.5. OOP Group Means Pertaining to Employment and Earnings by Sex (Percentages, Unless Stated Otherwise) 
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QOP enrollees and 489 controls. 




Table G.6. Control Group Means Pertaining to Employment and Earnings by Sex (Percentages, Unless Stated Otherwise) 
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QOP enrollees and 489 controls. 




Table G.7. OOP Group Means Pertaining to Risky Behaviors, Physical and Mental Well-Being, and Family Life by Sex (Percentages) 
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Note: Means were estimated using weights to adjust for differences between respondents and nonrespondents in baseline characteristics, response 

rates to the first and second telephone surveys, and outcomes measured in the in-person, first, and second telephone surveys. The evaluation 
sample had 580 QOP enrollees and 489 controls. 




Table G.8. Control Group Means Pertaining to Risky Behaviors, Physical and Mental Well-Being, and Family Life by Sex (Percentages) 
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Note: Means were estimated using weights to adjust for differences between respondents and nonrespondents in baseline characteristics, response 

rates to the first and second telephone surveys, and outcomes measured in the in-person, first, and second telephone surveys. The evaluation 
sample had 580 OOP enrollees and 489 controls. 




Table G.9. OOP Group Means Pertaining to High School Completion by Age When Entering Ninth 
Grade (Percentages) 







Means 




Outcome 


Age > 1 4 


Age < 1 4 


Total Sample 


Received HS diploma 


43 


70 


60 


Received HS diploma or GED 


61 


87 


78 



Source: Telephone surveys and transcripts. 

Note: Means were estimated using weights to adjust for differences between respondents and 

nonrespondents in baseline characteristics, response rates to the first and second telephone 
surveys, and outcomes measured in the in-person, first, and second telephone surveys. The 
evaluation sample had 580 QOP enrollees and 489 controls. 



Table G.10. Control Group Means Pertaining to High School Completion by Age When Entering 
Ninth Grade (Percentages) 







Means 




Outcome 


Age > 1 4 


Age < 1 4 


Total Sample 


Received HS diploma 


51 


63 


60 


Received HS diploma or GED 


61 


81 


75 



Source: Telephone surveys and transcripts. 

Note: Means were estimated using weights to adjust for differences between respondents and 

nonrespondents in baseline characteristics, response rates to the first and second telephone 
surveys, and outcomes measured in the in-person, first, and second telephone surveys. The 
evaluation sample had 580 QOP enrollees and 489 controls. 
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Table G.11. OOP Group Means Pertaining to Postsecondary Attainment by Age When Entering Ninth Grade (Percentages) 
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to the first and second telephone surveys, and outcomes measured in the in-person, first, and second telephone surveys. The evaluation sample 
had 580 QOP enrollees and 489 controls. 





Table G.12. Control Group Means Pertaining to Postsecondary Attainment by Age When Entering Ninth Grade (Percentages) 
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Note: Means were estimated using weights to adjust for differences between respondents and nonrespondents in baseline characteristics, response rates 

to the first and second telephone surveys, and outcomes measured in the in-person, first, and second telephone surveys. The evaluation sample 
had 580 QOP enrollees and 489 controls. 




Table G.13. OOP Group Means Pertaining to Employment and Earnings by Age When Entering Ninth Grade (Percentages, Unless Otherwise 
Indicated) 
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to the first and second telephone surveys, and outcomes measured in the in-person, first, and second telephone surveys. The evaluation sample 
had 580 QOP enrollees and 489 controls. 




Table G.14. Control Group Means Pertaining to Employment and Earnings by Age When Entering Ninth Grade (Percentages, Unless Otherwise 
Indicated) 
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to the first and second telephone surveys, and outcomes measured in the in-person, first, and second telephone surveys. The evaluation sample 
had 580 QOP enrollees and 489 controls. 





Table G.15. OOP Group Means Pertaining to Risky Behaviors, Physical and Mental Well-Being, and Family Life by Age When Entering Ninth Grade 
(Percentages) 
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Note: Means were estimated using weights to adjust for differences between respondents and nonrespondents in baseline characteristics, response rates to 

the first and second telephone surveys, and outcomes measured in the in-person, first, and second telephone surveys. The evaluation sample had 580 
OOP enrollees and 489 controls. 




Table G.16. Control Group Means Pertaining to Risky Behaviors, Physical and Mental Well-Being, and Family Life by Age When Entering Ninth 
Grade (Percentages) 
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Note: Means were estimated using weights to adjust for differences between respondents and nonrespondents in baseline characteristics, response rates to 

the first and second telephone surveys, and outcomes measured in the in-person, first, and second telephone surveys. The evaluation sample had 580 
OOP enrollees and 489 controls. 




Table G.17. OOP Group Means Pertaining to High School Completion by Rank in the Baseline Grade 
Distribution (Percentages) 

Means 



Outcome 


Bottom Third 


Middle Third 


Top Third 


Total Sample 


Received HS diploma 


44 


64 


74 


60 


Received HS diploma or GED 


69 


80 


86 


78 



Source: Telephone surveys and transcripts. 

Note: Means were estimated using weights to adjust for differences between respondents and nonrespondents 

in baseline characteristics, response rates to the first and second telephone surveys, and outcomes 
measured in the in-person, first, and second telephone surveys. The evaluation sample had 580 QOP 
enrollees and 489 controls. 



Table G.18. Control Group Means Pertaining to High School Completion by Rank in the Baseline Grade 
Distribution (Percentages) 

Means 



Outcome 


Bottom Third 


Middle Third 


Top Third 


Total Sample 


Received HS diploma 


46 


57 


78 


60 


Received HS diploma or GED 


67 


72 


90 


75 



Source: Telephone surveys and transcripts. 

Note: Means were estimated using weights to adjust for differences between respondents and nonrespondents 

in baseline characteristics, response rates to the first and second telephone surveys, and outcomes 
measured in the in-person, first, and second telephone surveys. The evaluation sample had 580 QOP 
enrollees and 489 controls. 
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Table G. 19 . OOP Group Means Pertaining to Postsecondary Attainment by Rank in the Baseline Grade Distribution (Percentages) 
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Note: Means were estimated using weights to adjust for differences between respondents and nonrespondents in baseline characteristics, response rates 

to the first and second telephone surveys, and outcomes measured in the in-person, first, and second telephone surveys. The evaluation sample 
had 580 QOP enrollees and 489 controls. 




Table G.20. Control Group Means Pertaining to Postsecondary Attainment by Rank in the Baseline Grade Distribution (Percentages) 
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Note: Means were estimated using weights to adjust for differences between respondents and nonrespondents in baseline characteristics, response rates 

to the first and second telephone surveys, and outcomes measured in the in-person, first, and second telephone surveys. The evaluation sample 
had 580 QOP enrollees and 489 controls. 
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Note: Means were estimated using weights to adjust for differences between respondents and nonrespondents in baseline characteristics, response rates 

to the first and second telephone surveys, and outcomes measured in the in-person, first, and second telephone surveys. The evaluation sample 
had 580 QOP enrollees and 489 controls. 
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Note: Means were estimated using weights to adjust for differences between respondents and nonrespondents in baseline characteristics, response rates 

to the first and second telephone surveys, and outcomes measured in the in-person, first, and second telephone surveys. The evaluation sample 
had 580 QOP enrollees and 489 controls. 
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Note: Means were estimated using weights to adjust for differences between respondents and nonrespondents in baseline characteristics, response rates to 

the first and second telephone surveys, and outcomes measured in the in-person, first, and second telephone surveys. The evaluation sample had 580 
QOP enrollees and 489 controls. 
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Note: Means were estimated using weights to adjust for differences between respondents and nonrespondents in baseline characteristics, response rates to 

the first and second telephone surveys, and outcomes measured in the in-person, first, and second telephone surveys. The evaluation sample had 580 
QOP enrollees and 489 controls. 
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Appendix H 

QOP AND Control Group Meansfor Sites and Site Subgroups 
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Table H.1. QOP Group Means Pertaining to High School Completion by Site (Percentages) 
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Note: Means were estimated using weights to adjust for differences between respondents and nonrespondents in baseline characteristics, response rates to 

the first and second telephone surveys, and outcomes measured in the in-person, first, and second telephone surveys. The evaluation sample had 580 
QOP enrollees and 489 controls. 




Table H.3. QOP Group Means Pertaining to Postsecondary Attainment by Site (Percentages) 
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Note: Means were estimated using weights to adjust for differences between respondents and nonrespondents in baseline characteristics, response rates to 

the first and second telephone surveys, and outcomes measured in the in-person, first, and second telephone surveys. The evaluation sample had 580 
QOP enrollees and 489 controls. 




Table H.4. Control Group Means Pertaining to Postsecondary Attainment by Site (Percentages) 
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Note: Means were estimated using weights to adjust for differences between respondents and nonrespondents in baseline characteristics, response rates to 

the first and second telephone surveys, and outcomes measured in the in-person, first, and second telephone surveys. The evaluation sample had 580 
QOP enrollees and 489 controls. 




Table H.5. OOP Group Means Pertaining to Employment and Earnings by Site (Percentages, Unless Stated Otherwise) 
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Note: Means were estimated using weights to adjust for differences between respondents and nonrespondents in baseline characteristics, response rates to 

the first and second telephone surveys, and outcomes measured in the in-person, first, and second telephone surveys. The evaluation sample had 580 
QOP enrollees and 489 controls. 




Table H.6. Control Group Means Pertaining to Employment and Earnings by Site (Percentages, Unless Stated Otherwise) 



_ _0> 
CC Q- 
O £ 
I— CC 
C/) 



oo in 

CD -1- -1- 



in 



CJ> 



1- 

OO CD 



o 

CD 



OO CO 

CM ir> 






CT> 

co' 



h- m 00 
CO 



(/) 

c 

CC 

CD 



CO 



CD 

■o 

CO 



O 

Q 



■o 

c 

_C0 

CD 

> 

_0> 

O 



o 

o 

"3 

O 



CM h- 1- 
h- CM 



OO 



00 00 CM 
in CM 1- 



O CT> 



I- CJ> o 
h- CM 



m CD CT> 
CD -1- -1- 



h- CT> m 
CM CM 



CJ> CD LO 

h- l- 



OO 



CM 

OO 



CM 



o 

CD 



00 



~0 T_ 
^ o 
o -Q 

^ CO 
Q. — 

E o 

CD ^ 
C 3 

3 o 

>^ >^ >^ 



it:; CO 

o = 

§1 

SI 

_0> t_ 

"o ^ 

CD 5 ^ 

y c-5- 

o ■- -C 

O CD 

■D ~ 

g'E 

c CO 
CD ^ 

^ CO 



cccc — 

0> 0> 0> 0> g 

3 3 3 -g 

O O O O w 



CD 

CT> 



CT> 



00 

CT> 



in 

CT> 



CM 

CT> 



00 

CT> 



CM 

CT> 



■D 

CD 

O 

Q. 



CD 

> 

LU 



CT> 

CD 



CO 

00 



CM 

CD 



O 

OO 



<J> 



CT> 



CO 

00 



CD -I- 
CD 



h- -I- 

m 1- 



<j> 

m 



CD 



CD 



CT> 

O 



CT> 

O CM 



(/) 

CO 

Q. 



■D 

^ CD 
Q. CD 

E 5 

CD ' 



CD 

E 

3 

c 

(/) 

c 

o 

E 



CM 



(/) 

^ CD 
CD O) 

CO > c 
Q. CD 

C E 

— CD CD 
■0 0)0. 
CD CO 
"c C/) 

O o £ 

Q. O c 

E 0) g 

LU CL E 



c 
o 

.o 

•= c 

(/) CD 

^ o 

2 to 

CD CD 

= ^ .cu 



(/) 

CO 

Q. 



00 

CM 



IT> 



CM 

CM 



CM 



m 

CO 



CM 



00 

ir> 



CD 

LD 



00 

in 



CO 



CO 

CD 



CD 

CD 



CD 

Q. 

■D 

CD 



Ie 

B g 

j- JT 

CD ^ 
^ (/) 
E 

5 .o 

c 4-- 

— c 

CO CD 

3 o 



CM 

in 



CD 

o 

CD 






CD 

cm" 



m 

o 

CO od 

CD 

CM 

co" 



in 

00 

o 



00 

CD 



o 

CM 

ir> 

cm" 



CM 



CD 

OO 



OO i- 

-I- 

CM 







E 


c 


E 


B 


'co 


CO 


o 


E 


— 




1 ■ 


o 




CO 




JO 


CD 


(/) 


_o^ 






CD 




c 




"c 


CD 




o 




0) 


E 


3 


Q. 


CM 


O 


E 




c 


3 




'co 


o 

jr 


(/) 

CO 

Q. 


E 


in 




"co 


CO 


C 


(/) 


• 4 —' 


if) 


O) 


(/) 

CO 

JO 

■*-- o 


O) 

c 

'c 


c 

'c 

CO 


CO 


CO 


CD 


(/) "c 


CD 




^ CD 


S 




1 § 


"o 

h- 


o 

X 



IT) 



1 - 00 
CM 



h- CD 
ID D 



D h- 



OO D 
D D 



O h- 
CO CM 



Tj- CD 

in D 



CD 

o 

c 

CO 

3 

(/) 

c 



c 

CD 



o ° 

B c/) 

c 

~o <i> 

rrt 



co _ 

CD CO 
^ Q. CO 



JO .Q ^ 

O O O (/) 

CO CO CO 

(/)(/)(/) c 
CO CO CO 05 

X X X JO 



Q> 

> 

3 

w 

Q> 

c 

o 

D 

Q. 

O 

O 



O 

CO 



H-7 



Note: Means were estimated using weights to adjust for differences between respondents and nonrespondents in baseline characteristics, response rates to 

the first and second telephone surveys, and outcomes measured in the in-person, first, and second telephone surveys. The evaluation sample had 580 
QOP enrollees and 489 controls. 
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Table H.9 OOP Group Means Pertaining to High School Completion by Site’s Deviation from 
QOP Model (Percentages) 







Means 






Sites that Deviated 


Sites that Deviated 






Substantially from 


Moderately from 




Outcome 


QOP Model 


QOP Model 


Total Sample 


Received HS diploma 


58 


61 


60 


Received HS diploma or GED 


74 


79 


78 



Source: Telephone surveys and transcripts. 

Note: Means were estimated using weights to adjust for differences between respondents and 

nonrespondents in baseline characteristics, response rates to the first and second telephone 
surveys, and outcomes measured in the in-person, first, and second telephone surveys. The 
evaluation sample had 580 QOP enrollees and 489 controls. 



Table H.10. Control Group Means Pertaining to High School Completion by Site’s Deviation from 
QOP Modei (Percentages) 







Means 




Outcome 


Sites that Deviated 
Substantially from 
QOP Model 


Sites that Deviated 
Moderately from QOP 
Model 


Total Sample 


Received HS diploma 


61 


59 


60 


Received HS diploma or GED 


76 


75 


75 



Source: Telephone surveys and transcripts. 

Note: Means were estimated using weights to adjust for differences between respondents and 

nonrespondents in baseline characteristics, response rates to the first and second telephone 
surveys, and outcomes measured in the in-person, first, and second telephone surveys. The 
evaluation sample had 580 QOP enrollees and 489 controls. 
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Table H.11. OOP Group Means Pertaining to Postsecondary Attainment by Site’s Deviation from OOP Model (Percentages) 
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Note: Means were estimated using weights to adjust for differences between respondents and nonrespondents in baseline characteristics, response rates 

to the first and second telephone surveys, and outcomes measured in the in-person, first, and second telephone surveys. The evaluation sample 
had 580 QOP enrollees and 489 controls. 





Table H.12. Control Group Means Pertaining to Postsecondary Attainment by Site’s Deviation from OOP Model (Percentages) 
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Note: Means were estimated using weights to adjust for differences between respondents and nonrespondents in baseline characteristics, response rates 

to the first and second telephone surveys, and outcomes measured in the in-person, first, and second telephone surveys. The evaluation sample 
had 580 QOP enrollees and 489 controls. 




Table H.13. OOP Group Means Pertaining to Employment and Earnings by Site’s Deviation from OOP Model (Percentages, Unless Stated 
Otherwise) 
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Note: Means were estimated using weights to adjust for differences between respondents and nonrespondents in baseline characteristics, response rates 

to the first and second telephone surveys, and outcomes measured in the in-person, first, and second telephone surveys. The evaluation sample 
had 580 QOP enrollees and 489 controls. 




Table H.14. Control Group Means Pertaining to Employment and Earnings by Site’s Deviation from OOP Model (Percentages, Unless Stated 
Otherwise) 
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Note: Means were estimated using weights to adjust for differences between respondents and nonrespondents in baseline characteristics, response rates 

to the first and second telephone surveys, and outcomes measured in the in-person, first, and second telephone surveys. The evaluation sample 
had 580 QOP enrollees and 489 controls. 




Table H.15. OOP Group Means Pertaining to Risky Behaviors, Physical and Mental Well-Being, and Family Life by Site’s Deviation from OOP Modei 
(Percentages) 
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Table H.16. Control Group Means Pertaining to Risky Behaviors, Physical and Mental Well-Being, and Family Life by Site’s Deviation from OOP 
Model (Percentages) 
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